[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:01 dist-test-slave-j8sq krb5kdc[11702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:01 dist-test-slave-j8sq krb5kdc[11702](info): set up 2 sockets
Sep 26 22:40:01 dist-test-slave-j8sq krb5kdc[11702](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 22:40:01.999960 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[11702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:40:02.117089 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:44391
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:44391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:02.196661 11715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:02.196873 11715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:02.196895 11715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:02.197993 11715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:02.198029 11715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:02.198043 11715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:02.198055 11715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:02.198067 11715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:02.199451 11715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:44391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:44391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:02.199635 11715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:02.199858 11715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:02.205132 11723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:02.205228 11721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:02.205195 11715 server_base.cc:1034] running on GCE node
W20260926 22:40:02.205374 11720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:02.205663 11715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:02.206030 11715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:02.207195 11715 hybrid_clock.cc:648] HybridClock initialized: now 1790462402207175 us; error 40 us; skew 500 ppm
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[11702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462402, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:02.209709 11715 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:40:02.210294 11715 webserver.cc:460] Webserver started at http://127.11.107.254:34851/ using document root <none> and password file <none>
I20260926 22:40:02.210537 11715 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:02.210606 11715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:02.210781 11715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:02.211896 11715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "9a81656f76a74ce18d0e14c77c260833"
format_stamp: "Formatted at 2026-09-26 22:40:02 on dist-test-slave-j8sq"
server_key: "08b5de49f5d3704d79f81199e3254b33"
server_key_iv: "099d37cf29174145ec25cc556cee529f"
server_key_version: "encryptionkey@0"
I20260926 22:40:02.212347 11715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "9a81656f76a74ce18d0e14c77c260833"
format_stamp: "Formatted at 2026-09-26 22:40:02 on dist-test-slave-j8sq"
server_key: "08b5de49f5d3704d79f81199e3254b33"
server_key_iv: "099d37cf29174145ec25cc556cee529f"
server_key_version: "encryptionkey@0"
I20260926 22:40:02.213743 11715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:02.214558 11730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:02.214758 11715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:02.214859 11715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "9a81656f76a74ce18d0e14c77c260833"
format_stamp: "Formatted at 2026-09-26 22:40:02 on dist-test-slave-j8sq"
server_key: "08b5de49f5d3704d79f81199e3254b33"
server_key_iv: "099d37cf29174145ec25cc556cee529f"
server_key_version: "encryptionkey@0"
I20260926 22:40:02.214984 11715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:02.236325 11715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:02.238431 11715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:02.238596 11715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:02.242432 11715 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:44391
I20260926 22:40:02.242468 11782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:44391 every 8 connection(s)
I20260926 22:40:02.242873 11715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:02.243696 11783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:02.246177 11783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833: Bootstrap starting.
I20260926 22:40:02.246743 11783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:02.247030 11783 log.cc:824] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:02.247699 11783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833: No bootstrap required, opened a new log
I20260926 22:40:02.249239 11783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a81656f76a74ce18d0e14c77c260833" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44391 } }
I20260926 22:40:02.249373 11783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:02.249396 11783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a81656f76a74ce18d0e14c77c260833, State: Initialized, Role: FOLLOWER
I20260926 22:40:02.249606 11783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a81656f76a74ce18d0e14c77c260833" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44391 } }
I20260926 22:40:02.249675 11783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:02.249696 11783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:02.249763 11783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:02.250298 11783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a81656f76a74ce18d0e14c77c260833" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44391 } }
I20260926 22:40:02.250406 11783 leader_election.cc:302] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a81656f76a74ce18d0e14c77c260833; no voters:
I20260926 22:40:02.250592 11783 leader_election.cc:288] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:02.250645 11786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:02.250809 11786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [term 1 LEADER]: Becoming Leader. State: Replica: 9a81656f76a74ce18d0e14c77c260833, State: Running, Role: LEADER
I20260926 22:40:02.250839 11783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:02.250900 11786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a81656f76a74ce18d0e14c77c260833" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44391 } }
I20260926 22:40:02.251289 11788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a81656f76a74ce18d0e14c77c260833. Latest consensus state: current_term: 1 leader_uuid: "9a81656f76a74ce18d0e14c77c260833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a81656f76a74ce18d0e14c77c260833" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44391 } } }
I20260926 22:40:02.251423 11787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a81656f76a74ce18d0e14c77c260833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a81656f76a74ce18d0e14c77c260833" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44391 } } }
I20260926 22:40:02.251435 11788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:02.251685 11787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:02.251791 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 11715
I20260926 22:40:02.251888 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:02.251726 11793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:02.252390 11793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:02.252674 11695 external_mini_cluster.cc:1428] Setting key 229ff463dff95a6753d23bb3c90f6119
I20260926 22:40:02.254774 11793 catalog_manager.cc:1347] Generated new cluster ID: c6d70f58f7a44d34a3f3733e31d71bb8
I20260926 22:40:02.254827 11793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[11702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:02.269577 11793 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:02.270212 11793 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:02.271703 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:02.253909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.11.107.254:44391)
0926 22:40:02.254242 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:02.254250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:02.254286 (+ 36us) server_negotiation.cc:407] Connection header received
0926 22:40:02.256332 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:02.256340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:02.256407 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:02.256564 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:02.258564 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.260782 (+ 2218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.262367 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.263027 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.263138 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:02.267517 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:02.267522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:02.267522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:02.267533 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:02.269589 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:02.270122 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:02.270125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:02.270125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:02.270248 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:02.270577 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:02.270579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:02.270579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:02.270875 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:02.271071 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:02.271324 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 22:40:02.271472 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":76,"threads_started":1}
I20260926 22:40:02.276719 11793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a81656f76a74ce18d0e14c77c260833: Generated new TSK 0
I20260926 22:40:02.276880 11793 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:40:02.305032 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44391
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:02.387739 11809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:02.387982 11809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:02.388023 11809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:02.389178 11809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:02.389250 11809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:02.389299 11809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:02.390817 11809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44391
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:02.391104 11809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:02.391343 11809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:02.396332 11815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:02.396410 11809 server_base.cc:1034] running on GCE node
W20260926 22:40:02.396327 11817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:02.396364 11814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:02.396781 11809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:02.397037 11809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:02.398216 11809 hybrid_clock.cc:648] HybridClock initialized: now 1790462402398202 us; error 27 us; skew 500 ppm
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[11702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462402, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:02.400627 11809 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:40:02.401028 11809 webserver.cc:460] Webserver started at http://127.11.107.193:42797/ using document root <none> and password file <none>
I20260926 22:40:02.401325 11809 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:02.401373 11809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:02.401460 11809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:02.402513 11809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "73200cd001c74ea7930324484813e0ce"
format_stamp: "Formatted at 2026-09-26 22:40:02 on dist-test-slave-j8sq"
server_key: "98ef7349cfd570ccd5f4e07e94f41c3b"
server_key_iv: "e3a51286b8beee622a38488b8f8ecc34"
server_key_version: "encryptionkey@0"
I20260926 22:40:02.402809 11809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "73200cd001c74ea7930324484813e0ce"
format_stamp: "Formatted at 2026-09-26 22:40:02 on dist-test-slave-j8sq"
server_key: "98ef7349cfd570ccd5f4e07e94f41c3b"
server_key_iv: "e3a51286b8beee622a38488b8f8ecc34"
server_key_version: "encryptionkey@0"
I20260926 22:40:02.404198 11809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:02.404894 11824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:02.405112 11809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:02.405224 11809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "73200cd001c74ea7930324484813e0ce"
format_stamp: "Formatted at 2026-09-26 22:40:02 on dist-test-slave-j8sq"
server_key: "98ef7349cfd570ccd5f4e07e94f41c3b"
server_key_iv: "e3a51286b8beee622a38488b8f8ecc34"
server_key_version: "encryptionkey@0"
I20260926 22:40:02.405303 11809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:02.412061 11809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:02.414022 11809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:02.414186 11809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:02.414420 11809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:02.414739 11809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:02.414795 11809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:02.414836 11809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:02.414862 11809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:02.420833 11809 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:36955
I20260926 22:40:02.420883 11937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:36955 every 8 connection(s)
I20260926 22:40:02.421198 11809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[11702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462402, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:02.430043 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 11809
I20260926 22:40:02.430150 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:02.430212 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:02.421922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:37103 (local address 127.11.107.254:44391)
0926 22:40:02.421988 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:02.421990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:02.422178 (+ 188us) server_negotiation.cc:407] Connection header received
0926 22:40:02.422682 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:02.422683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:02.422693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:02.422746 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:02.423765 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.424260 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.425258 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.425447 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.425500 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:02.427559 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:02.427560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:02.427561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:02.427568 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:02.428783 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:02.429212 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:02.429213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:02.429214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:02.429273 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:02.429513 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:02.429514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:02.429514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:02.429624 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:02.429703 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:02.430030 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 22:40:02.430120 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 22:40:02.430326 11695 external_mini_cluster.cc:1428] Setting key b2c55963e5ff5ae6ffdeca54bede3611
I20260926 22:40:02.430389 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:02.421914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44391 (local address 127.11.107.193:37103)
0926 22:40:02.422074 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:02.422109 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:02.422579 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:02.422797 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:02.422800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:02.423492 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:02.423687 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:02.423692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.424360 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.424361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:02.425178 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:02.425180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.425520 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.425521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:02.425627 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:02.426126 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:02.426134 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:02.427462 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:02.428868 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:02.428875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:02.428877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:02.429110 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:02.429340 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:02.429341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:02.429342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:02.429445 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:02.429705 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:02.429711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:02.429884 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 22:40:02.430008 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 22:40:02.430170 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260926 22:40:02.431087 11938 heartbeater.cc:347] Connected to a master server at 127.11.107.254:44391
I20260926 22:40:02.431188 11938 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:02.431380 11938 heartbeater.cc:511] Master 127.11.107.254:44391 requested a full tablet report, sending...
I20260926 22:40:02.431849 11746 ts_manager.cc:194] Registered new tserver with Master: 73200cd001c74ea7930324484813e0ce (127.11.107.193:36955)
I20260926 22:40:02.432444 11746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:37103
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:40:02.454749 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44391
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:02.533248 11943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:02.533452 11943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:02.533473 11943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:02.534782 11943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:02.534835 11943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:02.534862 11943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:02.536322 11943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44391
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:02.536576 11943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:02.536839 11943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:02.541790 11948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:02.541828 11949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:02.541805 11951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:02.541961 11943 server_base.cc:1034] running on GCE node
I20260926 22:40:02.542289 11943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:02.542539 11943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:02.543715 11943 hybrid_clock.cc:648] HybridClock initialized: now 1790462402543681 us; error 48 us; skew 500 ppm
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[11702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462402, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:02.546396 11943 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:40:02.546818 11943 webserver.cc:460] Webserver started at http://127.11.107.194:40963/ using document root <none> and password file <none>
I20260926 22:40:02.547053 11943 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:02.547130 11943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:02.547276 11943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:02.548331 11943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "94418d176ad245b483dccbe04abf7e87"
format_stamp: "Formatted at 2026-09-26 22:40:02 on dist-test-slave-j8sq"
server_key: "95af2da3aea71697a2a35294462c2ae4"
server_key_iv: "afe3831882c70e9208f43a391966b2ee"
server_key_version: "encryptionkey@0"
I20260926 22:40:02.548692 11943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "94418d176ad245b483dccbe04abf7e87"
format_stamp: "Formatted at 2026-09-26 22:40:02 on dist-test-slave-j8sq"
server_key: "95af2da3aea71697a2a35294462c2ae4"
server_key_iv: "afe3831882c70e9208f43a391966b2ee"
server_key_version: "encryptionkey@0"
I20260926 22:40:02.550038 11943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:40:02.550812 11958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:02.550977 11943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:02.551084 11943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "94418d176ad245b483dccbe04abf7e87"
format_stamp: "Formatted at 2026-09-26 22:40:02 on dist-test-slave-j8sq"
server_key: "95af2da3aea71697a2a35294462c2ae4"
server_key_iv: "afe3831882c70e9208f43a391966b2ee"
server_key_version: "encryptionkey@0"
I20260926 22:40:02.551156 11943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:02.563304 11943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:02.565521 11943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:02.565685 11943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:02.565939 11943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:02.566244 11943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:02.566298 11943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:02.566334 11943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:02.566359 11943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:02.572043 11943 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:43167
I20260926 22:40:02.572084 12071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:43167 every 8 connection(s)
I20260926 22:40:02.572428 11943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[11702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462402, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:02.579054 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 11943
I20260926 22:40:02.579171 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:02.579342 11695 external_mini_cluster.cc:1428] Setting key bf850789848d3cbd888978be6c0600ce
I20260926 22:40:02.581575 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:02.573128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:35403 (local address 127.11.107.254:44391)
0926 22:40:02.573188 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:02.573190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:02.573376 (+ 186us) server_negotiation.cc:407] Connection header received
0926 22:40:02.573939 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:02.573940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:02.573953 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:02.574015 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:02.574907 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.575404 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.576401 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.576615 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.576687 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:02.578688 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:02.578689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:02.578690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:02.578697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:02.580268 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:02.580644 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:02.580646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:02.580646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:02.580707 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:02.580918 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:02.580919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:02.580919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:02.581056 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:02.581125 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:02.581409 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 22:40:02.581487 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:40:02.581792 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:02.573111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44391 (local address 127.11.107.194:35403)
0926 22:40:02.573263 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:02.573296 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:02.573794 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:02.574089 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:02.574092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:02.574631 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:02.574808 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:02.574812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.575499 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.575500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:02.576287 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:02.576289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.576704 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.576705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:02.576823 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:02.577252 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:02.577260 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:02.578593 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:02.580349 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:02.580354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:02.580356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:02.580566 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:02.580768 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:02.580769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:02.580769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:02.580860 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:02.581119 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:02.581123 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:02.581229 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 22:40:02.581381 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 22:40:02.581533 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260926 22:40:02.582239 12072 heartbeater.cc:347] Connected to a master server at 127.11.107.254:44391
I20260926 22:40:02.582341 12072 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:02.582536 12072 heartbeater.cc:511] Master 127.11.107.254:44391 requested a full tablet report, sending...
I20260926 22:40:02.582875 11746 ts_manager.cc:194] Registered new tserver with Master: 94418d176ad245b483dccbe04abf7e87 (127.11.107.194:43167)
I20260926 22:40:02.583257 11746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:35403
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:40:02.605209 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44391
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:02.686846 12077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:02.687078 12077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:02.687119 12077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:02.688263 12077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:02.688328 12077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:02.688373 12077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:02.689800 12077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44391
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:02.690119 12077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:02.690346 12077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:02.695294 12083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:02.695266 12085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:02.695313 12082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:02.695515 12077 server_base.cc:1034] running on GCE node
I20260926 22:40:02.695691 12077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:02.695922 12077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:02.697113 12077 hybrid_clock.cc:648] HybridClock initialized: now 1790462402697103 us; error 41 us; skew 500 ppm
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[11702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462402, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:02.699668 12077 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:40:02.700098 12077 webserver.cc:460] Webserver started at http://127.11.107.195:37259/ using document root <none> and password file <none>
I20260926 22:40:02.700320 12077 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:02.700382 12077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:02.700531 12077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:02.701548 12077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "5f5c77d578b44e6fa68a2ff0029517aa"
format_stamp: "Formatted at 2026-09-26 22:40:02 on dist-test-slave-j8sq"
server_key: "c297f86de54e0b7db68b52e8dd7a5d21"
server_key_iv: "d1104980910781e2bc0ea9166a3f0875"
server_key_version: "encryptionkey@0"
I20260926 22:40:02.702026 12077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "5f5c77d578b44e6fa68a2ff0029517aa"
format_stamp: "Formatted at 2026-09-26 22:40:02 on dist-test-slave-j8sq"
server_key: "c297f86de54e0b7db68b52e8dd7a5d21"
server_key_iv: "d1104980910781e2bc0ea9166a3f0875"
server_key_version: "encryptionkey@0"
I20260926 22:40:02.703341 12077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:02.704046 12092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:02.704216 12077 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:02.704309 12077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "5f5c77d578b44e6fa68a2ff0029517aa"
format_stamp: "Formatted at 2026-09-26 22:40:02 on dist-test-slave-j8sq"
server_key: "c297f86de54e0b7db68b52e8dd7a5d21"
server_key_iv: "d1104980910781e2bc0ea9166a3f0875"
server_key_version: "encryptionkey@0"
I20260926 22:40:02.704392 12077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:02.723694 12077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:02.726178 12077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:02.726348 12077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:02.726599 12077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:02.726912 12077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:02.726964 12077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:02.726999 12077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:02.727046 12077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:02.732712 12077 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:41527
I20260926 22:40:02.732792 12205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:41527 every 8 connection(s)
I20260926 22:40:02.733107 12077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[11702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462402, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:02.740291 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 12077
I20260926 22:40:02.740398 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:02.740597 11695 external_mini_cluster.cc:1428] Setting key e8bdd247cf6421579ca178c2f750770b
I20260926 22:40:02.742885 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:02.733831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:40285 (local address 127.11.107.254:44391)
0926 22:40:02.733919 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:02.733921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:02.734103 (+ 182us) server_negotiation.cc:407] Connection header received
0926 22:40:02.734712 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:02.734714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:02.734726 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:02.734785 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:02.735787 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.736358 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.737342 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.737530 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.737602 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:02.739814 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:02.739816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:02.739817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:02.739823 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:02.741384 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:02.741930 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:02.741931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:02.741931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:02.741989 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:02.742259 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:02.742260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:02.742261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:02.742378 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:02.742452 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:02.742725 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 22:40:02.742799 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 22:40:02.743072 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:02.733825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44391 (local address 127.11.107.195:40285)
0926 22:40:02.733985 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:02.734020 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:02.734601 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:02.734864 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:02.734867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:02.735488 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:02.735680 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:02.735684 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.736447 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.736448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:02.737239 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:02.737241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.737619 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.737620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:02.737739 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:02.738252 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:02.738261 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:02.739711 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:02.741463 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:02.741469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:02.741471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:02.741830 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:02.742068 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:02.742069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:02.742070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:02.742185 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:02.742444 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:02.742451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:02.742585 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 22:40:02.742683 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 22:40:02.742835 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20260926 22:40:02.743496 12206 heartbeater.cc:347] Connected to a master server at 127.11.107.254:44391
I20260926 22:40:02.743569 12206 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:02.743757 12206 heartbeater.cc:511] Master 127.11.107.254:44391 requested a full tablet report, sending...
I20260926 22:40:02.744107 11746 ts_manager.cc:194] Registered new tserver with Master: 5f5c77d578b44e6fa68a2ff0029517aa (127.11.107.195:41527)
I20260926 22:40:02.744493 11746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:40285
I20260926 22:40:02.752657 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[11702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462402, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[11702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462402, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.107.193@KRBTEST.COM
I20260926 22:40:02.776224 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:02.767713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.11.107.193:36955)
0926 22:40:02.767888 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:02.767894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:02.767905 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:40:02.767938 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:02.767940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:02.767953 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:02.768040 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:02.768626 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.769406 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.770279 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.770630 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.770714 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:02.773233 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:02.773242 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:02.773242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:02.773251 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:02.774930 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:02.775345 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:02.775349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:02.775349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:02.775413 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:02.775620 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:02.775621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:02.775621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:02.775778 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:02.775973 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:02.776017 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 22:40:02.776103 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
W20260926 22:40:02.776669 11852 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43388
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[11702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[11702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.193@KRBTEST.COM
I20260926 22:40:02.799435 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:02.792373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.11.107.193:36955)
0926 22:40:02.792446 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:02.792448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:02.792522 (+ 74us) server_negotiation.cc:407] Connection header received
0926 22:40:02.792592 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:02.792593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:02.792604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:02.792654 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:02.793230 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.793720 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.794488 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:02.794688 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:02.794740 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:02.796973 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:02.796974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:02.796975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:02.796981 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:02.798457 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:02.798800 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:02.798801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:02.798801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:02.798850 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:02.799018 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:02.799018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:02.799018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:02.799103 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:02.799196 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:02.799286 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 22:40:02.799345 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 22:40:02.800100 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 11809
I20260926 22:40:02.805876 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 11943
I20260926 22:40:02.810678 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 12077
I20260926 22:40:02.814762 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 11715
2026-09-26T22:40:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1872 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[12220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[12220](info): set up 2 sockets
Sep 26 22:40:02 dist-test-slave-j8sq krb5kdc[12220](info): commencing operation
krb5kdc: starting...
W20260926 22:40:03.846383 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:03 dist-test-slave-j8sq krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:40:03.961910 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:38187
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:43185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:38187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:04.041335 12233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:04.041566 12233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:04.041603 12233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:04.042833 12233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:04.042898 12233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:04.042922 12233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:04.042951 12233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:04.042981 12233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:04.044423 12233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:43185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:38187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:38187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:04.044704 12233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:04.044929 12233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:04.049530 12241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:04.049569 12239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:04.049535 12238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:04.049750 12233 server_base.cc:1034] running on GCE node
I20260926 22:40:04.050009 12233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:04.050294 12233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:04.051441 12233 hybrid_clock.cc:648] HybridClock initialized: now 1790462404051429 us; error 27 us; skew 500 ppm
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462404, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:04.054015 12233 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:40:04.054437 12233 webserver.cc:460] Webserver started at http://127.11.107.254:38063/ using document root <none> and password file <none>
I20260926 22:40:04.054668 12233 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:04.054736 12233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:04.054888 12233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:04.055986 12233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "774698fac38a480aaef7450c48cbf124"
format_stamp: "Formatted at 2026-09-26 22:40:04 on dist-test-slave-j8sq"
server_key: "2419d07d7a8a9495df85840b54570bb7"
server_key_iv: "11beaafb0626aae9c1bf987147bd5c79"
server_key_version: "encryptionkey@0"
I20260926 22:40:04.056346 12233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "774698fac38a480aaef7450c48cbf124"
format_stamp: "Formatted at 2026-09-26 22:40:04 on dist-test-slave-j8sq"
server_key: "2419d07d7a8a9495df85840b54570bb7"
server_key_iv: "11beaafb0626aae9c1bf987147bd5c79"
server_key_version: "encryptionkey@0"
I20260926 22:40:04.057593 12233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:40:04.058364 12248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:04.058521 12233 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:04.058619 12233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "774698fac38a480aaef7450c48cbf124"
format_stamp: "Formatted at 2026-09-26 22:40:04 on dist-test-slave-j8sq"
server_key: "2419d07d7a8a9495df85840b54570bb7"
server_key_iv: "11beaafb0626aae9c1bf987147bd5c79"
server_key_version: "encryptionkey@0"
I20260926 22:40:04.058703 12233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:04.075491 12233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:04.077752 12233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:04.077950 12233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:04.081408 12233 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:38187
I20260926 22:40:04.081483 12300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:38187 every 8 connection(s)
I20260926 22:40:04.081805 12233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:04.082625 12301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:04.084885 12301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124: Bootstrap starting.
I20260926 22:40:04.085471 12301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:04.085753 12301 log.cc:824] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:04.086234 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 12233
I20260926 22:40:04.086319 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:04.086470 11695 external_mini_cluster.cc:1428] Setting key 0e33fa5750a0bebff5afae217e7d219d
I20260926 22:40:04.086537 12301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124: No bootstrap required, opened a new log
I20260926 22:40:04.088373 12301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774698fac38a480aaef7450c48cbf124" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38187 } }
I20260926 22:40:04.088527 12301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:04.088580 12301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774698fac38a480aaef7450c48cbf124, State: Initialized, Role: FOLLOWER
I20260926 22:40:04.088707 12301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774698fac38a480aaef7450c48cbf124" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38187 } }
I20260926 22:40:04.088785 12301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:04.088827 12301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:04.088894 12301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:04.089406 12301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774698fac38a480aaef7450c48cbf124" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38187 } }
I20260926 22:40:04.089516 12301 leader_election.cc:302] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 774698fac38a480aaef7450c48cbf124; no voters:
I20260926 22:40:04.089708 12301 leader_election.cc:288] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:04.089730 12306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:04.089854 12306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [term 1 LEADER]: Becoming Leader. State: Replica: 774698fac38a480aaef7450c48cbf124, State: Running, Role: LEADER
I20260926 22:40:04.089933 12306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774698fac38a480aaef7450c48cbf124" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38187 } }
I20260926 22:40:04.089982 12301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:04.091706 12308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 774698fac38a480aaef7450c48cbf124. Latest consensus state: current_term: 1 leader_uuid: "774698fac38a480aaef7450c48cbf124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774698fac38a480aaef7450c48cbf124" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38187 } } }
I20260926 22:40:04.091799 12308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:04.092131 12307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "774698fac38a480aaef7450c48cbf124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774698fac38a480aaef7450c48cbf124" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38187 } } }
I20260926 22:40:04.092269 12307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124 [sys.catalog]: This master's current role is: LEADER
W20260926 22:40:04.092604 12320 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:40:04.092664 12320 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:40:04.092756 12321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:04.092990 12321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:04.094614 12321 catalog_manager.cc:1347] Generated new cluster ID: 59ca3badc0014f1da0afaa622bf7546b
I20260926 22:40:04.094677 12321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:04.097658 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.086971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.11.107.254:38187)
0926 22:40:04.087157 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:04.087166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:04.087198 (+ 32us) server_negotiation.cc:407] Connection header received
0926 22:40:04.087727 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:04.087733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:04.087777 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:04.087918 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:04.088516 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.089956 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.090861 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.091495 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.091922 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.094519 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:04.094522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:04.094523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:04.094534 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:04.096228 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.096598 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.096601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.096602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.096690 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.096898 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.096898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.096899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.097110 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:04.097281 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:04.097343 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 22:40:04.097469 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260926 22:40:04.115932 12321 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:04.116612 12321 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:04.122076 12321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 774698fac38a480aaef7450c48cbf124: Generated new TSK 0
I20260926 22:40:04.122263 12321 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:40:04.156023 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38187
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:43185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:04.258690 12327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:04.258881 12327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:04.258903 12327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:04.259967 12327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:04.260000 12327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:04.260025 12327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:04.261437 12327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:43185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38187
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:04.261615 12327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:04.261787 12327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:04.266780 12335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:04.266865 12327 server_base.cc:1034] running on GCE node
W20260926 22:40:04.266795 12332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:04.266829 12333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:04.267192 12327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:04.267450 12327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:04.268635 12327 hybrid_clock.cc:648] HybridClock initialized: now 1790462404268601 us; error 43 us; skew 500 ppm
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462404, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:04.271170 12327 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:40:04.271646 12327 webserver.cc:460] Webserver started at http://127.11.107.193:35091/ using document root <none> and password file <none>
I20260926 22:40:04.271878 12327 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:04.271945 12327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:04.272099 12327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:04.273096 12327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "b2f1a5b95101443a9b12ffb9b11af78e"
format_stamp: "Formatted at 2026-09-26 22:40:04 on dist-test-slave-j8sq"
server_key: "7b3d8811ad7bea85e87160cd071973f5"
server_key_iv: "30173bf5c4e02883ddbcc100ab9bba39"
server_key_version: "encryptionkey@0"
I20260926 22:40:04.273447 12327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "b2f1a5b95101443a9b12ffb9b11af78e"
format_stamp: "Formatted at 2026-09-26 22:40:04 on dist-test-slave-j8sq"
server_key: "7b3d8811ad7bea85e87160cd071973f5"
server_key_iv: "30173bf5c4e02883ddbcc100ab9bba39"
server_key_version: "encryptionkey@0"
I20260926 22:40:04.274809 12327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:04.275496 12342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:04.275656 12327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:04.275751 12327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "b2f1a5b95101443a9b12ffb9b11af78e"
format_stamp: "Formatted at 2026-09-26 22:40:04 on dist-test-slave-j8sq"
server_key: "7b3d8811ad7bea85e87160cd071973f5"
server_key_iv: "30173bf5c4e02883ddbcc100ab9bba39"
server_key_version: "encryptionkey@0"
I20260926 22:40:04.275830 12327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:04.290109 12327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:04.292192 12327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:04.292373 12327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:04.292596 12327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:04.292887 12327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:04.292940 12327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:04.292974 12327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:04.293000 12327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:04.298645 12327 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:38165
I20260926 22:40:04.298695 12455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:38165 every 8 connection(s)
I20260926 22:40:04.299011 12327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:04.301203 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 12327
I20260926 22:40:04.301278 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:04.301435 11695 external_mini_cluster.cc:1428] Setting key 5117a23b8751c0afc25b4ae72d3359df
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462404, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:04.308521 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.299674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:57565 (local address 127.11.107.254:38187)
0926 22:40:04.299754 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:04.299757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:04.299926 (+ 169us) server_negotiation.cc:407] Connection header received
0926 22:40:04.300842 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:04.300843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:04.300853 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:04.300904 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:04.301548 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.302263 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.303239 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.303453 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.303558 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.305603 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:04.305605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:04.305605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:04.305614 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:04.307229 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.307606 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.307607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.307607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.307669 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.307895 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.307896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.307896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.308033 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:04.308115 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:04.308336 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 22:40:04.308425 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 22:40:04.308542 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.299649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:38187 (local address 127.11.107.193:57565)
0926 22:40:04.299812 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:04.299843 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:04.300750 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:04.300956 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:04.300959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:04.301192 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:04.301427 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.301432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.302351 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.302352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.303141 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.303143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.303568 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.303569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.303705 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.304198 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:04.304205 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:04.305506 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:04.307309 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:04.307314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:04.307316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:04.307529 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:04.307738 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:04.307738 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:04.307739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:04.307829 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:04.308099 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:04.308102 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:04.308211 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 22:40:04.308289 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 22:40:04.308397 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260926 22:40:04.309015 12456 heartbeater.cc:347] Connected to a master server at 127.11.107.254:38187
I20260926 22:40:04.309125 12456 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:04.309307 12456 heartbeater.cc:511] Master 127.11.107.254:38187 requested a full tablet report, sending...
I20260926 22:40:04.309883 12265 ts_manager.cc:194] Registered new tserver with Master: b2f1a5b95101443a9b12ffb9b11af78e (127.11.107.193:38165)
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
I20260926 22:40:04.310858 12265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:57565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:40:04.328184 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38187
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:43185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:04.407928 12461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:04.408120 12461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:04.408143 12461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:04.409227 12461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:04.409260 12461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:04.409286 12461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:04.410703 12461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:43185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38187
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:04.410878 12461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:04.411043 12461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:04.415903 12469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:04.415933 12467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:04.415943 12466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:04.415966 12461 server_base.cc:1034] running on GCE node
I20260926 22:40:04.416338 12461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:04.416574 12461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:04.417742 12461 hybrid_clock.cc:648] HybridClock initialized: now 1790462404417725 us; error 30 us; skew 500 ppm
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462404, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:04.420132 12461 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:40:04.420480 12461 webserver.cc:460] Webserver started at http://127.11.107.194:42237/ using document root <none> and password file <none>
I20260926 22:40:04.420658 12461 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:04.420696 12461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:04.420784 12461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:04.421685 12461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9"
format_stamp: "Formatted at 2026-09-26 22:40:04 on dist-test-slave-j8sq"
server_key: "92ea60edb9383b5acce665e9fd93de76"
server_key_iv: "f09ed63577b19140ec28b70f20adbc3e"
server_key_version: "encryptionkey@0"
I20260926 22:40:04.422045 12461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9"
format_stamp: "Formatted at 2026-09-26 22:40:04 on dist-test-slave-j8sq"
server_key: "92ea60edb9383b5acce665e9fd93de76"
server_key_iv: "f09ed63577b19140ec28b70f20adbc3e"
server_key_version: "encryptionkey@0"
I20260926 22:40:04.423267 12461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:04.423920 12476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:04.424147 12461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:04.424224 12461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9"
format_stamp: "Formatted at 2026-09-26 22:40:04 on dist-test-slave-j8sq"
server_key: "92ea60edb9383b5acce665e9fd93de76"
server_key_iv: "f09ed63577b19140ec28b70f20adbc3e"
server_key_version: "encryptionkey@0"
I20260926 22:40:04.424274 12461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:04.447561 12461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:04.449736 12461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:04.449863 12461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:04.450028 12461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:04.450344 12461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:04.450390 12461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:04.450459 12461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:04.450510 12461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:04.456050 12461 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:43907
I20260926 22:40:04.456126 12589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:43907 every 8 connection(s)
I20260926 22:40:04.456426 12461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462404, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:04.462781 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 12461
I20260926 22:40:04.462888 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:04.463053 11695 external_mini_cluster.cc:1428] Setting key b8c04ac793121170e6cc4fc3d7b9f45c
I20260926 22:40:04.465641 12592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.457085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:38187 (local address 127.11.107.194:35077)
0926 22:40:04.457251 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:04.457290 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:04.458207 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:04.458415 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:04.458418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:04.458654 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:04.458846 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.458850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.459469 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.459470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.460261 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.460262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.460606 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.460607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.460755 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.461139 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:04.461146 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:04.462561 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:04.464181 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:04.464199 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:04.464201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:04.464544 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:04.464728 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:04.464729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:04.464729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:04.464865 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:04.465165 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:04.465169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:04.465300 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 22:40:04.465384 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 22:40:04.465494 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260926 22:40:04.465631 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.457125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:35077 (local address 127.11.107.254:38187)
0926 22:40:04.457189 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:04.457191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:04.457374 (+ 183us) server_negotiation.cc:407] Connection header received
0926 22:40:04.458293 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:04.458294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:04.458303 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:04.458361 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:04.458914 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.459395 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.460332 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.460506 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.460553 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.462641 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:04.462642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:04.462642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:04.462649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:04.464099 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.464618 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.464619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.464619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.464666 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.464942 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.464943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.464943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.465082 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:04.465165 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:04.465433 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 22:40:04.465516 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 22:40:04.466197 12590 heartbeater.cc:347] Connected to a master server at 127.11.107.254:38187
I20260926 22:40:04.466274 12590 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:04.466444 12590 heartbeater.cc:511] Master 127.11.107.254:38187 requested a full tablet report, sending...
I20260926 22:40:04.466814 12265 ts_manager.cc:194] Registered new tserver with Master: 8e1336f646ea4eeb855bbed3a5e4c4e9 (127.11.107.194:43907)
I20260926 22:40:04.467351 12265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:35077
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:40:04.486733 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38187
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:43185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:04.565747 12595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:04.566013 12595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:04.566039 12595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:04.567135 12595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:04.567169 12595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:04.567195 12595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:04.568603 12595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:43185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38187
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:04.568790 12595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:04.568948 12595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:04.573665 12601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:04.573740 12595 server_base.cc:1034] running on GCE node
W20260926 22:40:04.573701 12603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:04.573657 12600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:04.574066 12595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:04.574295 12595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:04.575444 12595 hybrid_clock.cc:648] HybridClock initialized: now 1790462404575423 us; error 36 us; skew 500 ppm
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462404, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:04.577692 12595 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:40:04.578136 12595 webserver.cc:460] Webserver started at http://127.11.107.195:36741/ using document root <none> and password file <none>
I20260926 22:40:04.578354 12595 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:04.578440 12595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:04.578588 12595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:04.579658 12595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "e8a05cd83e5349d39e0d70c6b21ae65b"
format_stamp: "Formatted at 2026-09-26 22:40:04 on dist-test-slave-j8sq"
server_key: "0fcead19f2ba16c5529b5f01170a5470"
server_key_iv: "49aebadf62819b6f22f83ded33f92f5e"
server_key_version: "encryptionkey@0"
I20260926 22:40:04.580029 12595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "e8a05cd83e5349d39e0d70c6b21ae65b"
format_stamp: "Formatted at 2026-09-26 22:40:04 on dist-test-slave-j8sq"
server_key: "0fcead19f2ba16c5529b5f01170a5470"
server_key_iv: "49aebadf62819b6f22f83ded33f92f5e"
server_key_version: "encryptionkey@0"
I20260926 22:40:04.581324 12595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:04.582049 12610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:04.582223 12595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:04.582321 12595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "e8a05cd83e5349d39e0d70c6b21ae65b"
format_stamp: "Formatted at 2026-09-26 22:40:04 on dist-test-slave-j8sq"
server_key: "0fcead19f2ba16c5529b5f01170a5470"
server_key_iv: "49aebadf62819b6f22f83ded33f92f5e"
server_key_version: "encryptionkey@0"
I20260926 22:40:04.582417 12595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:04.594748 12595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:04.596439 12595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:04.596593 12595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:04.596805 12595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:04.597090 12595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:04.597142 12595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:04.597182 12595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:04.597209 12595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:04.603117 12595 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:35797
I20260926 22:40:04.603158 12723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:35797 every 8 connection(s)
I20260926 22:40:04.603448 12595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462404, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:04.610973 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 12595
I20260926 22:40:04.611073 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:04.611250 11695 external_mini_cluster.cc:1428] Setting key 25e48733d8903cef78b1752b3d207e5a
I20260926 22:40:04.612717 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.604086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:56311 (local address 127.11.107.254:38187)
0926 22:40:04.604152 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:04.604154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:04.604371 (+ 217us) server_negotiation.cc:407] Connection header received
0926 22:40:04.605276 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:04.605277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:04.605286 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:04.605330 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:04.606061 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.606581 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.607522 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.607710 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.607769 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.609710 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:04.609711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:04.609712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:04.609718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:04.611288 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.611780 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.611781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.611781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.611854 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.612127 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.612127 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.612128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.612261 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:04.612313 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:04.612597 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 22:40:04.612657 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:40:04.612882 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.604058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:38187 (local address 127.11.107.195:56311)
0926 22:40:04.604231 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:04.604272 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:04.605172 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:04.605416 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:04.605419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:04.605650 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:04.605931 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.605937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.606660 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.606661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.607440 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.607441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.607797 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.607798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.607888 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.608322 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:04.608334 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:04.609633 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:04.611376 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:04.611383 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:04.611385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:04.611713 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:04.611924 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:04.611925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:04.611925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:04.612067 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:04.612327 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:04.612333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:04.612483 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 22:40:04.612571 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 22:40:04.612683 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
I20260926 22:40:04.613312 12724 heartbeater.cc:347] Connected to a master server at 127.11.107.254:38187
I20260926 22:40:04.613422 12724 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:04.613595 12724 heartbeater.cc:511] Master 127.11.107.254:38187 requested a full tablet report, sending...
I20260926 22:40:04.613973 12265 ts_manager.cc:194] Registered new tserver with Master: e8a05cd83e5349d39e0d70c6b21ae65b (127.11.107.195:35797)
I20260926 22:40:04.614289 12265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:56311
I20260926 22:40:04.622957 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:40:04.630160 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.624378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.11.107.254:38187)
0926 22:40:04.624465 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:04.624467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:04.624560 (+ 93us) server_negotiation.cc:407] Connection header received
0926 22:40:04.624657 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:04.624658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:04.624667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:04.624723 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:04.625258 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.625731 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.626490 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.626687 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.626790 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.627855 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:04.627857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:04.627857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:04.627864 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:04.629006 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.629339 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.629340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.629340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.629393 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.629606 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.629607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.629607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.629757 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:04.629888 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:04.629994 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 22:40:04.630069 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 22:40:04.632145 12265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:04.633622 12265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:04.639983 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.636032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.11.107.193:38165)
0926 22:40:04.636248 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:04.636252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:04.636367 (+ 115us) server_negotiation.cc:407] Connection header received
0926 22:40:04.636443 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:04.636445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:04.636460 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:04.636533 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:04.636756 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.637461 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.639102 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.639570 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.639659 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.639715 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:04.639852 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 22:40:04.639898 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260926 22:40:04.640051 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.635777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.11.107.195:35797)
0926 22:40:04.636095 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:04.636099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:04.636109 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:40:04.636145 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:04.636147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:04.636164 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:04.636256 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:04.636388 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.637527 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.638994 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.639682 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.639782 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.639833 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:04.639939 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 22:40:04.639978 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":74,"threads_started":1}
I20260926 22:40:04.640053 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.635736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:43907 (local address 127.0.0.1:52252)
0926 22:40:04.636004 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:04.636013 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:04.636082 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:04.636641 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:04.636642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:04.636644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:04.636699 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.636701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.637883 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.637884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.638868 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.638870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.639780 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.639781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.639843 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.639847 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:04.639877 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 22:40:04.639969 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260926 22:40:04.640308 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.635919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:38165 (local address 127.0.0.1:54298)
0926 22:40:04.636295 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:04.636303 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:04.636367 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:04.636618 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:04.636619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:04.636621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:04.636672 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.636674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.637554 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.637554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.639021 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.639023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.639658 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.639659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.639732 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.639747 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 22:40:04.640090 (+ 343us) client_negotiation.cc:240] Negotiation successful
0926 22:40:04.640252 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":39,"thread_start_us":62,"threads_started":1}
I20260926 22:40:04.640345 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.635833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.11.107.194:43907)
0926 22:40:04.636484 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:04.636488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:04.636497 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:04.636533 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:04.636535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:04.636549 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:04.636622 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:04.636781 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.637776 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.638955 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.639673 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.640179 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.640240 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:04.640261 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 22:40:04.640293 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":72,"threads_started":1}
I20260926 22:40:04.640682 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.635707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:35797 (local address 127.0.0.1:34902)
0926 22:40:04.635856 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:04.635891 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:04.635956 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:04.636260 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:04.636263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:04.636267 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:04.636329 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.636331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.637632 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.637633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.638927 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.638929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.639790 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.639791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.639863 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.639866 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:04.639916 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 22:40:04.639987 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":66,"thread_start_us":50,"threads_started":1}
I20260926 22:40:04.641021 12390 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6364289bb84dfa9a8e30a23b2e7532 (DEFAULT_TABLE table=test-table [id=b95458718ddb4f4bbe43500557c78792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:04.641022 12524 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6364289bb84dfa9a8e30a23b2e7532 (DEFAULT_TABLE table=test-table [id=b95458718ddb4f4bbe43500557c78792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:04.641359 12524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6364289bb84dfa9a8e30a23b2e7532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:04.641274 12658 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6364289bb84dfa9a8e30a23b2e7532 (DEFAULT_TABLE table=test-table [id=b95458718ddb4f4bbe43500557c78792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:04.641357 12390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6364289bb84dfa9a8e30a23b2e7532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:04.641548 12658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6364289bb84dfa9a8e30a23b2e7532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:04.643927 12741 tablet_bootstrap.cc:492] T 8c6364289bb84dfa9a8e30a23b2e7532 P b2f1a5b95101443a9b12ffb9b11af78e: Bootstrap starting.
I20260926 22:40:04.644555 12741 tablet_bootstrap.cc:654] T 8c6364289bb84dfa9a8e30a23b2e7532 P b2f1a5b95101443a9b12ffb9b11af78e: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:04.644619 12743 tablet_bootstrap.cc:492] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9: Bootstrap starting.
I20260926 22:40:04.645187 12743 tablet_bootstrap.cc:654] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:04.645268 12741 log.cc:824] T 8c6364289bb84dfa9a8e30a23b2e7532 P b2f1a5b95101443a9b12ffb9b11af78e: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:04.645764 12743 log.cc:824] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:04.645990 12742 tablet_bootstrap.cc:492] T 8c6364289bb84dfa9a8e30a23b2e7532 P e8a05cd83e5349d39e0d70c6b21ae65b: Bootstrap starting.
I20260926 22:40:04.646116 12741 tablet_bootstrap.cc:492] T 8c6364289bb84dfa9a8e30a23b2e7532 P b2f1a5b95101443a9b12ffb9b11af78e: No bootstrap required, opened a new log
I20260926 22:40:04.646202 12741 ts_tablet_manager.cc:1397] T 8c6364289bb84dfa9a8e30a23b2e7532 P b2f1a5b95101443a9b12ffb9b11af78e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:40:04.646597 12742 tablet_bootstrap.cc:654] T 8c6364289bb84dfa9a8e30a23b2e7532 P e8a05cd83e5349d39e0d70c6b21ae65b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:04.646615 12743 tablet_bootstrap.cc:492] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9: No bootstrap required, opened a new log
I20260926 22:40:04.646677 12743 ts_tablet_manager.cc:1397] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 22:40:04.647173 12742 log.cc:824] T 8c6364289bb84dfa9a8e30a23b2e7532 P e8a05cd83e5349d39e0d70c6b21ae65b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:04.647917 12742 tablet_bootstrap.cc:492] T 8c6364289bb84dfa9a8e30a23b2e7532 P e8a05cd83e5349d39e0d70c6b21ae65b: No bootstrap required, opened a new log
I20260926 22:40:04.647986 12742 ts_tablet_manager.cc:1397] T 8c6364289bb84dfa9a8e30a23b2e7532 P e8a05cd83e5349d39e0d70c6b21ae65b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:04.648018 12741 raft_consensus.cc:348] T 8c6364289bb84dfa9a8e30a23b2e7532 P b2f1a5b95101443a9b12ffb9b11af78e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a05cd83e5349d39e0d70c6b21ae65b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35797 } } peers { permanent_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43907 } } peers { permanent_uuid: "b2f1a5b95101443a9b12ffb9b11af78e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38165 } }
I20260926 22:40:04.648204 12741 raft_consensus.cc:374] T 8c6364289bb84dfa9a8e30a23b2e7532 P b2f1a5b95101443a9b12ffb9b11af78e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:04.648249 12741 raft_consensus.cc:729] T 8c6364289bb84dfa9a8e30a23b2e7532 P b2f1a5b95101443a9b12ffb9b11af78e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f1a5b95101443a9b12ffb9b11af78e, State: Initialized, Role: FOLLOWER
I20260926 22:40:04.648404 12741 consensus_queue.cc:260] T 8c6364289bb84dfa9a8e30a23b2e7532 P b2f1a5b95101443a9b12ffb9b11af78e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a05cd83e5349d39e0d70c6b21ae65b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35797 } } peers { permanent_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43907 } } peers { permanent_uuid: "b2f1a5b95101443a9b12ffb9b11af78e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38165 } }
I20260926 22:40:04.648693 12456 heartbeater.cc:503] Master 127.11.107.254:38187 was elected leader, sending a full tablet report...
I20260926 22:40:04.648744 12743 raft_consensus.cc:348] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a05cd83e5349d39e0d70c6b21ae65b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35797 } } peers { permanent_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43907 } } peers { permanent_uuid: "b2f1a5b95101443a9b12ffb9b11af78e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38165 } }
I20260926 22:40:04.648913 12743 raft_consensus.cc:374] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:04.648693 12741 ts_tablet_manager.cc:1428] T 8c6364289bb84dfa9a8e30a23b2e7532 P b2f1a5b95101443a9b12ffb9b11af78e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:40:04.648948 12743 raft_consensus.cc:729] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1336f646ea4eeb855bbed3a5e4c4e9, State: Initialized, Role: FOLLOWER
I20260926 22:40:04.649068 12743 consensus_queue.cc:260] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a05cd83e5349d39e0d70c6b21ae65b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35797 } } peers { permanent_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43907 } } peers { permanent_uuid: "b2f1a5b95101443a9b12ffb9b11af78e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38165 } }
I20260926 22:40:04.649323 12590 heartbeater.cc:503] Master 127.11.107.254:38187 was elected leader, sending a full tablet report...
I20260926 22:40:04.649314 12743 ts_tablet_manager.cc:1428] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 22:40:04.650079 12742 raft_consensus.cc:348] T 8c6364289bb84dfa9a8e30a23b2e7532 P e8a05cd83e5349d39e0d70c6b21ae65b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a05cd83e5349d39e0d70c6b21ae65b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35797 } } peers { permanent_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43907 } } peers { permanent_uuid: "b2f1a5b95101443a9b12ffb9b11af78e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38165 } }
I20260926 22:40:04.650206 12742 raft_consensus.cc:374] T 8c6364289bb84dfa9a8e30a23b2e7532 P e8a05cd83e5349d39e0d70c6b21ae65b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:04.650269 12742 raft_consensus.cc:729] T 8c6364289bb84dfa9a8e30a23b2e7532 P e8a05cd83e5349d39e0d70c6b21ae65b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a05cd83e5349d39e0d70c6b21ae65b, State: Initialized, Role: FOLLOWER
I20260926 22:40:04.650384 12742 consensus_queue.cc:260] T 8c6364289bb84dfa9a8e30a23b2e7532 P e8a05cd83e5349d39e0d70c6b21ae65b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a05cd83e5349d39e0d70c6b21ae65b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35797 } } peers { permanent_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43907 } } peers { permanent_uuid: "b2f1a5b95101443a9b12ffb9b11af78e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38165 } }
I20260926 22:40:04.650619 12742 ts_tablet_manager.cc:1428] T 8c6364289bb84dfa9a8e30a23b2e7532 P e8a05cd83e5349d39e0d70c6b21ae65b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:40:04.650658 12724 heartbeater.cc:503] Master 127.11.107.254:38187 was elected leader, sending a full tablet report...
I20260926 22:40:04.691314 12748 raft_consensus.cc:482] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:04.691490 12748 raft_consensus.cc:504] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a05cd83e5349d39e0d70c6b21ae65b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35797 } } peers { permanent_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43907 } } peers { permanent_uuid: "b2f1a5b95101443a9b12ffb9b11af78e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38165 } }
I20260926 22:40:04.691857 12748 leader_election.cc:288] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8a05cd83e5349d39e0d70c6b21ae65b (127.11.107.195:35797), b2f1a5b95101443a9b12ffb9b11af78e (127.11.107.193:38165)
I20260926 22:40:04.695086 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.692080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:45383 (local address 127.11.107.195:35797)
0926 22:40:04.692138 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:04.692140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:04.692149 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:04.692184 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:04.692185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:04.692195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:04.692280 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:04.692454 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.692984 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.694158 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.694710 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.694784 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.694792 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:04.694983 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 22:40:04.695020 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:40:04.695086 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.692085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:54741 (local address 127.11.107.193:38165)
0926 22:40:04.692138 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:04.692140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:04.692327 (+ 187us) server_negotiation.cc:407] Connection header received
0926 22:40:04.692363 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:04.692364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:04.692374 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:04.692435 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:04.692580 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.693046 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.694129 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.694709 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.694784 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.694792 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:04.694983 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 22:40:04.695020 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 22:40:04.695118 12592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.692017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:35797 (local address 127.11.107.194:45383)
0926 22:40:04.692058 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:04.692070 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:04.692110 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:04.692290 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:04.692291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:04.692293 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:04.692383 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.692385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.693066 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.693067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.694052 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.694055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.694800 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.694801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.694894 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.694896 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:40:04.694970 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 22:40:04.695016 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":124}
I20260926 22:40:04.695128 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.692017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:38165 (local address 127.11.107.194:54741)
0926 22:40:04.692244 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:04.692255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:04.692317 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:04.692453 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:04.692454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:04.692455 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:04.692518 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.692519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.693129 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.693129 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.694023 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:04.694025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.694800 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.694801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:04.694890 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.694895 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:40:04.694970 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 22:40:04.695010 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":48,"thread_start_us":110,"threads_started":1}
I20260926 22:40:04.695467 12678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6364289bb84dfa9a8e30a23b2e7532" candidate_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a05cd83e5349d39e0d70c6b21ae65b" is_pre_election: true
I20260926 22:40:04.695664 12678 raft_consensus.cc:2454] T 8c6364289bb84dfa9a8e30a23b2e7532 P e8a05cd83e5349d39e0d70c6b21ae65b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1336f646ea4eeb855bbed3a5e4c4e9 in term 0.
I20260926 22:40:04.695753 12410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6364289bb84dfa9a8e30a23b2e7532" candidate_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f1a5b95101443a9b12ffb9b11af78e" is_pre_election: true
I20260926 22:40:04.695902 12410 raft_consensus.cc:2454] T 8c6364289bb84dfa9a8e30a23b2e7532 P b2f1a5b95101443a9b12ffb9b11af78e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1336f646ea4eeb855bbed3a5e4c4e9 in term 0.
I20260926 22:40:04.695933 12478 leader_election.cc:302] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e1336f646ea4eeb855bbed3a5e4c4e9, e8a05cd83e5349d39e0d70c6b21ae65b; no voters:
I20260926 22:40:04.696081 12748 raft_consensus.cc:2781] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:04.696139 12748 raft_consensus.cc:482] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:04.696168 12748 raft_consensus.cc:3037] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:04.696885 12748 raft_consensus.cc:504] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a05cd83e5349d39e0d70c6b21ae65b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35797 } } peers { permanent_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43907 } } peers { permanent_uuid: "b2f1a5b95101443a9b12ffb9b11af78e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38165 } }
I20260926 22:40:04.697034 12748 leader_election.cc:288] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [CANDIDATE]: Term 1 election: Requested vote from peers e8a05cd83e5349d39e0d70c6b21ae65b (127.11.107.195:35797), b2f1a5b95101443a9b12ffb9b11af78e (127.11.107.193:38165)
I20260926 22:40:04.697194 12678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6364289bb84dfa9a8e30a23b2e7532" candidate_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a05cd83e5349d39e0d70c6b21ae65b"
I20260926 22:40:04.697206 12410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6364289bb84dfa9a8e30a23b2e7532" candidate_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f1a5b95101443a9b12ffb9b11af78e"
I20260926 22:40:04.697295 12410 raft_consensus.cc:3037] T 8c6364289bb84dfa9a8e30a23b2e7532 P b2f1a5b95101443a9b12ffb9b11af78e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:04.697281 12678 raft_consensus.cc:3037] T 8c6364289bb84dfa9a8e30a23b2e7532 P e8a05cd83e5349d39e0d70c6b21ae65b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:04.698057 12410 raft_consensus.cc:2454] T 8c6364289bb84dfa9a8e30a23b2e7532 P b2f1a5b95101443a9b12ffb9b11af78e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e1336f646ea4eeb855bbed3a5e4c4e9 in term 1.
I20260926 22:40:04.698197 12678 raft_consensus.cc:2454] T 8c6364289bb84dfa9a8e30a23b2e7532 P e8a05cd83e5349d39e0d70c6b21ae65b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e1336f646ea4eeb855bbed3a5e4c4e9 in term 1.
I20260926 22:40:04.698238 12479 leader_election.cc:302] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e1336f646ea4eeb855bbed3a5e4c4e9, b2f1a5b95101443a9b12ffb9b11af78e; no voters:
I20260926 22:40:04.698346 12748 raft_consensus.cc:2781] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:04.698546 12748 raft_consensus.cc:686] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [term 1 LEADER]: Becoming Leader. State: Replica: 8e1336f646ea4eeb855bbed3a5e4c4e9, State: Running, Role: LEADER
I20260926 22:40:04.698642 12748 consensus_queue.cc:237] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a05cd83e5349d39e0d70c6b21ae65b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35797 } } peers { permanent_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43907 } } peers { permanent_uuid: "b2f1a5b95101443a9b12ffb9b11af78e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38165 } }
I20260926 22:40:04.699518 12264 catalog_manager.cc:5495] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e1336f646ea4eeb855bbed3a5e4c4e9 (127.11.107.194). New cstate: current_term: 1 leader_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a05cd83e5349d39e0d70c6b21ae65b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e1336f646ea4eeb855bbed3a5e4c4e9" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2f1a5b95101443a9b12ffb9b11af78e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38165 } health_report { overall_health: UNKNOWN } } }
W20260926 22:40:04.706944 12591 tablet.cc:2369] T 8c6364289bb84dfa9a8e30a23b2e7532 P 8e1336f646ea4eeb855bbed3a5e4c4e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.193@KRBTEST.COM
I20260926 22:40:04.716848 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.710192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54304 (local address 127.11.107.193:38165)
0926 22:40:04.710234 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:04.710236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:04.710317 (+ 81us) server_negotiation.cc:407] Connection header received
0926 22:40:04.710352 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:04.710352 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:04.710360 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:04.710406 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:04.710931 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.711400 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.712058 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.712250 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.712298 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.714553 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:04.714561 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:04.714561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:04.714567 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:04.715924 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.716228 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.716230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.716230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.716281 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.716440 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.716440 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.716440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.716560 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:04.716656 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:04.716727 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 22:40:04.716786 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.107.193@KRBTEST.COM
I20260926 22:40:04.739295 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.732647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.11.107.193:38165)
0926 22:40:04.732729 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:04.732730 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:04.732774 (+ 44us) server_negotiation.cc:407] Connection header received
0926 22:40:04.732818 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:04.732819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:04.732828 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:04.732883 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:04.733396 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.733924 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.734598 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.734780 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.734838 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.736842 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:04.736843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:04.736843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:04.736849 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:04.738250 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.738612 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.738613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.738613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.738695 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.738865 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.738865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.738865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.738952 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:04.739021 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:04.739143 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 22:40:04.739198 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.193@KRBTEST.COM
I20260926 22:40:04.760425 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:04.753823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.11.107.193:38165)
0926 22:40:04.753901 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:04.753903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:04.753991 (+ 88us) server_negotiation.cc:407] Connection header received
0926 22:40:04.754077 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:04.754078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:04.754088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:04.754173 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:04.754704 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.755168 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.755902 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:04.756086 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:04.756175 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:04.758218 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:04.758219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:04.758219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:04.758226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:04.759382 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.759707 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.759708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.759708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.759766 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:04.759953 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:04.759954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:04.759954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:04.760048 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:04.760149 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:04.760253 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 22:40:04.760326 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 22:40:04.760905 12370 tablet_service.cc:2908] Rejecting scan request for tablet 8c6364289bb84dfa9a8e30a23b2e7532: Uninitialized: clean time has not yet been initialized
I20260926 22:40:04.761813 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 12327
I20260926 22:40:04.766436 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 12461
I20260926 22:40:04.770865 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 12595
I20260926 22:40:04.775063 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 12233
2026-09-26T22:40:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1960 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12765](info): set up 2 sockets
Sep 26 22:40:04 dist-test-slave-j8sq krb5kdc[12765](info): commencing operation
krb5kdc: starting...
W20260926 22:40:05.806349 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:05 dist-test-slave-j8sq krb5kdc[12765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:40:05.920593 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:40007
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:40007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:05.998411 12778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:05.998656 12778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:05.998703 12778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:05.999854 12778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:05.999928 12778 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 22:40:05.999949 12778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:05.999970 12778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:06.000000 12778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:06.000023 12778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:06.001438 12778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:40007
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:40007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:06.001719 12778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:06.001956 12778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:06.006628 12784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:06.006639 12778 server_base.cc:1034] running on GCE node
W20260926 22:40:06.006623 12786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:06.006623 12783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:06.007228 12778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:06.007471 12778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:06.008610 12778 hybrid_clock.cc:648] HybridClock initialized: now 1790462406008590 us; error 38 us; skew 500 ppm
Sep 26 22:40:06 dist-test-slave-j8sq krb5kdc[12765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462406, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:06.015072 12778 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:40:06.015555 12778 webserver.cc:460] Webserver started at http://127.11.107.254:42317/ using document root <none> and password file <none>
I20260926 22:40:06.015807 12778 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:06.015854 12778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:06.016018 12778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:06.017057 12778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "1f0371c5c489407ebf3efad44792f728"
format_stamp: "Formatted at 2026-09-26 22:40:06 on dist-test-slave-j8sq"
server_key: "0c77f181efc9b8edb18b7f648e4c91d4"
server_key_iv: "f18c05fcc41db36538293c1ab6602f30"
server_key_version: "encryptionkey@0"
I20260926 22:40:06.017418 12778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "1f0371c5c489407ebf3efad44792f728"
format_stamp: "Formatted at 2026-09-26 22:40:06 on dist-test-slave-j8sq"
server_key: "0c77f181efc9b8edb18b7f648e4c91d4"
server_key_iv: "f18c05fcc41db36538293c1ab6602f30"
server_key_version: "encryptionkey@0"
I20260926 22:40:06.018801 12778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:06.019497 12793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:06.019694 12778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:06.019779 12778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "1f0371c5c489407ebf3efad44792f728"
format_stamp: "Formatted at 2026-09-26 22:40:06 on dist-test-slave-j8sq"
server_key: "0c77f181efc9b8edb18b7f648e4c91d4"
server_key_iv: "f18c05fcc41db36538293c1ab6602f30"
server_key_version: "encryptionkey@0"
I20260926 22:40:06.019851 12778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:06.049705 12778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:06.051985 12778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:06.052143 12778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:06.057127 12778 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:40007
I20260926 22:40:06.057181 12855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:40007 every 8 connection(s)
I20260926 22:40:06.057505 12778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:06.058122 12856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:06.060375 12856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728: Bootstrap starting.
I20260926 22:40:06.060953 12856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:06.061239 12856 log.cc:824] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:06.061986 12856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728: No bootstrap required, opened a new log
I20260926 22:40:06.063989 12856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0371c5c489407ebf3efad44792f728" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 40007 } }
I20260926 22:40:06.064168 12856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:06.064216 12856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0371c5c489407ebf3efad44792f728, State: Initialized, Role: FOLLOWER
I20260926 22:40:06.064375 12856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0371c5c489407ebf3efad44792f728" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 40007 } }
I20260926 22:40:06.064455 12856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:06.064497 12856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:06.064544 12856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:06.065078 12856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0371c5c489407ebf3efad44792f728" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 40007 } }
I20260926 22:40:06.065191 12856 leader_election.cc:302] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f0371c5c489407ebf3efad44792f728; no voters:
I20260926 22:40:06.065219 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 12778
I20260926 22:40:06.065291 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:06.065354 12856 leader_election.cc:288] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:06.065416 12859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:06.065469 11695 external_mini_cluster.cc:1428] Setting key 265ddbabc5e392c79ba1554ea466bbfe
I20260926 22:40:06.065577 12859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0371c5c489407ebf3efad44792f728, State: Running, Role: LEADER
I20260926 22:40:06.065608 12856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:06.065689 12859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0371c5c489407ebf3efad44792f728" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 40007 } }
I20260926 22:40:06.066015 12859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f0371c5c489407ebf3efad44792f728. Latest consensus state: current_term: 1 leader_uuid: "1f0371c5c489407ebf3efad44792f728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0371c5c489407ebf3efad44792f728" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 40007 } } }
I20260926 22:40:06.066094 12859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:06.066025 12860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f0371c5c489407ebf3efad44792f728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0371c5c489407ebf3efad44792f728" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 40007 } } }
I20260926 22:40:06.066313 12860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:06.066417 12868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:06.066994 12868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:06.068761 12868 catalog_manager.cc:1347] Generated new cluster ID: 9f83689d52984f0eaf1767e047dc74ea
I20260926 22:40:06.068809 12868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:40:06 dist-test-slave-j8sq krb5kdc[12765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:06.077224 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.066092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.11.107.254:40007)
0926 22:40:06.066309 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.066319 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.066520 (+ 201us) server_negotiation.cc:407] Connection header received
0926 22:40:06.066651 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.066658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.066724 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.066892 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:06.067644 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.068996 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.070054 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.070642 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.070738 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.073646 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:06.073649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:06.073650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:06.073661 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:06.075641 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.076036 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.076039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.076040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.076149 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.076367 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.076368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.076368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.076638 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:06.076785 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.076893 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.077014 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":99,"threads_started":1}
I20260926 22:40:06.079088 12868 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:06.079766 12868 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:06.084332 12868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728: Generated new TSK 0
I20260926 22:40:06.084486 12868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:40:06.119024 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:40007
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:06.209095 12882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:06.209277 12882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:06.209299 12882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:06.210529 12882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:06.210569 12882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:06.210589 12882 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:40:06.210632 12882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:06.212047 12882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.107.254:40007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:06.212225 12882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:06.212581 12882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:06.217569 12890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:06.217610 12888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:06.217540 12887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:06.217756 12882 server_base.cc:1034] running on GCE node
I20260926 22:40:06.217980 12882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:06.218210 12882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:06.219347 12882 hybrid_clock.cc:648] HybridClock initialized: now 1790462406219337 us; error 28 us; skew 500 ppm
Sep 26 22:40:06 dist-test-slave-j8sq krb5kdc[12765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462406, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:06.221589 12882 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:40:06.221995 12882 webserver.cc:460] Webserver started at http://127.11.107.193:39013/ using document root <none> and password file <none>
I20260926 22:40:06.222221 12882 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:06.222261 12882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:06.222345 12882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:06.223320 12882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "bd74331259c74083ba60c3798a062874"
format_stamp: "Formatted at 2026-09-26 22:40:06 on dist-test-slave-j8sq"
server_key: "3e01b93b935938a5233b325ea61fbe9a"
server_key_iv: "9b99c847a2143a90a33bb58fa3842c64"
server_key_version: "encryptionkey@0"
I20260926 22:40:06.223655 12882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "bd74331259c74083ba60c3798a062874"
format_stamp: "Formatted at 2026-09-26 22:40:06 on dist-test-slave-j8sq"
server_key: "3e01b93b935938a5233b325ea61fbe9a"
server_key_iv: "9b99c847a2143a90a33bb58fa3842c64"
server_key_version: "encryptionkey@0"
I20260926 22:40:06.224884 12882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:40:06.225534 12897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:06.225679 12882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:06.225767 12882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "bd74331259c74083ba60c3798a062874"
format_stamp: "Formatted at 2026-09-26 22:40:06 on dist-test-slave-j8sq"
server_key: "3e01b93b935938a5233b325ea61fbe9a"
server_key_iv: "9b99c847a2143a90a33bb58fa3842c64"
server_key_version: "encryptionkey@0"
I20260926 22:40:06.225817 12882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:06.243674 12882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:06.245366 12882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:06.245520 12882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:06.246057 12882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:06.246095 12882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:06.246117 12882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:06.246130 12882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:06.252542 12882 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:36615
I20260926 22:40:06.252570 13012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:36615 every 8 connection(s)
I20260926 22:40:06.252928 12882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
Sep 26 22:40:06 dist-test-slave-j8sq krb5kdc[12765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462406, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:06.254017 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 12882
I20260926 22:40:06.254145 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:06.254334 11695 external_mini_cluster.cc:1428] Setting key 142b9311b973128f091118748c3594b0
I20260926 22:40:06.256556 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.246541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:33187 (local address 127.11.107.254:40007)
0926 22:40:06.246624 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.246627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.246760 (+ 133us) server_negotiation.cc:407] Connection header received
0926 22:40:06.247223 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.247224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.247237 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.247314 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:06.248123 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.248964 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.250514 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.250864 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.250949 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.253520 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:06.253521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:06.253521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:06.253531 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:06.255192 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.255604 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.255605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.255606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.255665 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.255921 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.255922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.255922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.256041 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:06.256118 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.256385 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.256468 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 22:40:06.256754 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.246470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:40007 (local address 127.11.107.193:33187)
0926 22:40:06.246631 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.246670 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.247118 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.247376 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.247379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.247737 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:06.248007 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.248014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.249063 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.249064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.250414 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.250416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.250951 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.250951 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.251057 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.251591 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:06.251602 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:06.253432 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:06.255277 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.255284 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.255286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.255511 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.255738 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.255738 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.255739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.255851 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.256108 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:06.256114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:06.256254 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.256363 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.256518 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260926 22:40:06.258157 13013 heartbeater.cc:347] Connected to a master server at 127.11.107.254:40007
I20260926 22:40:06.258255 13013 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:06.258416 13013 heartbeater.cc:511] Master 127.11.107.254:40007 requested a full tablet report, sending...
I20260926 22:40:06.258888 12809 ts_manager.cc:194] Registered new tserver with Master: bd74331259c74083ba60c3798a062874 (127.11.107.193:36615)
I20260926 22:40:06.259482 12809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:33187
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
I20260926 22:40:06.265018 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.258286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:57125 (local address 127.11.107.254:40007)
0926 22:40:06.258326 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.258328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.258514 (+ 186us) server_negotiation.cc:407] Connection header received
0926 22:40:06.258588 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.258589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.258599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.258647 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:06.260056 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.260642 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.261300 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.261503 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.261579 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.262469 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:06.262470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:06.262471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:06.262477 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:06.263824 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.264207 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.264208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.264209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.264268 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.264493 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.264494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.264494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.264609 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:06.264701 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.264858 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.264929 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:40:06.265033 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.258240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:40007 (local address 127.11.107.193:57125)
0926 22:40:06.258451 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.258461 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.258538 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.259695 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.259696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.259899 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:06.259962 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.259963 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.260737 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.260738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.261220 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.261221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.261577 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.261578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.261656 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.262081 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:06.262085 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:06.262387 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:06.263926 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.263927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.263928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.264134 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.264334 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.264335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.264335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.264393 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.264694 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:06.264695 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:06.264739 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.264830 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.264930 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:40:06.278646 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:40007
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:06.360039 13022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:06.360260 13022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:06.360286 13022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:06.361414 13022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:06.361457 13022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:06.361479 13022 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:40:06.361507 13022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:06.362979 13022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.107.254:40007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:06.363173 13022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:06.363350 13022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:06.368539 13030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:06.368554 13028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:06.368558 13027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:06.368727 13022 server_base.cc:1034] running on GCE node
I20260926 22:40:06.368929 13022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:06.369174 13022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:06.370350 13022 hybrid_clock.cc:648] HybridClock initialized: now 1790462406370336 us; error 24 us; skew 500 ppm
Sep 26 22:40:06 dist-test-slave-j8sq krb5kdc[12765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462406, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:06.372772 13022 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:40:06.373137 13022 webserver.cc:460] Webserver started at http://127.11.107.194:43443/ using document root <none> and password file <none>
I20260926 22:40:06.373327 13022 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:06.373366 13022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:06.373453 13022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:06.374506 13022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "9d26d9be0d7e41239388dda163b1523e"
format_stamp: "Formatted at 2026-09-26 22:40:06 on dist-test-slave-j8sq"
server_key: "a50c00d64d86c53023c38537edba59bd"
server_key_iv: "c4d5ffa62107ec444a86029d20b58320"
server_key_version: "encryptionkey@0"
I20260926 22:40:06.374809 13022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "9d26d9be0d7e41239388dda163b1523e"
format_stamp: "Formatted at 2026-09-26 22:40:06 on dist-test-slave-j8sq"
server_key: "a50c00d64d86c53023c38537edba59bd"
server_key_iv: "c4d5ffa62107ec444a86029d20b58320"
server_key_version: "encryptionkey@0"
I20260926 22:40:06.376000 13022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:06.376652 13037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:06.376832 13022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:06.376904 13022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "9d26d9be0d7e41239388dda163b1523e"
format_stamp: "Formatted at 2026-09-26 22:40:06 on dist-test-slave-j8sq"
server_key: "a50c00d64d86c53023c38537edba59bd"
server_key_iv: "c4d5ffa62107ec444a86029d20b58320"
server_key_version: "encryptionkey@0"
I20260926 22:40:06.376955 13022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:06.384347 13022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:06.386049 13022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:06.386157 13022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:06.386574 13022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:06.386613 13022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:06.386634 13022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:06.386647 13022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:06.393157 13022 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:36645
I20260926 22:40:06.393213 13152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:36645 every 8 connection(s)
Sep 26 22:40:06 dist-test-slave-j8sq krb5kdc[12765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462406, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:06.393510 13022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:40:06.396261 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.387036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:37035 (local address 127.11.107.254:40007)
0926 22:40:06.387075 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.387078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.387276 (+ 198us) server_negotiation.cc:407] Connection header received
0926 22:40:06.387773 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.387774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.387788 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.387838 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:06.388615 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.389409 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.391114 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.391399 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.391469 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.394093 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:06.394094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:06.394095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:06.394102 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:06.395190 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.395587 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.395587 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.395588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.395634 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.395804 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.395805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.395805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.395895 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:06.395944 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.396145 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.396200 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:40:06.396483 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.387028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:40007 (local address 127.11.107.194:37035)
0926 22:40:06.387168 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.387200 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.387654 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.387903 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.387907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.388245 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:06.388523 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.388529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.389516 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.389517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.390990 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.390993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.391501 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.391502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.391682 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.392263 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:06.392275 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:06.393989 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:06.395256 (+ 1267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.395261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.395262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.395522 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.395676 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.395677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.395677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.395757 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.395950 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:06.395955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:06.396055 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.396132 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.396239 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260926 22:40:06.397011 13153 heartbeater.cc:347] Connected to a master server at 127.11.107.254:40007
I20260926 22:40:06.397104 13153 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:06.397275 13153 heartbeater.cc:511] Master 127.11.107.254:40007 requested a full tablet report, sending...
I20260926 22:40:06.397603 12810 ts_manager.cc:194] Registered new tserver with Master: 9d26d9be0d7e41239388dda163b1523e (127.11.107.194:36645)
I20260926 22:40:06.398162 12810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:37035
I20260926 22:40:06.402882 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.398053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:33067 (local address 127.11.107.254:40007)
0926 22:40:06.398100 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.398101 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.398151 (+ 50us) server_negotiation.cc:407] Connection header received
0926 22:40:06.398236 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.398237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.398247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.398302 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:06.398643 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.399101 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.399783 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.399976 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.400053 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.400786 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:06.400787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:06.400787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:06.400793 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:06.401787 (+ 994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.402140 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.402141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.402141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.402189 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.402390 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.402391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.402391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.402517 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:06.402588 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.402736 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.402800 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:40:06.403049 13159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.397973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:40007 (local address 127.11.107.194:33067)
0926 22:40:06.398093 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.398103 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.398160 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.398308 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.398309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.398479 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:06.398526 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.398527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.399189 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.399190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.399692 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.399694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.400056 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.400056 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.400128 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.400442 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:06.400445 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:06.400702 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:06.401902 (+ 1200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.401903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.401903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.402060 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.402259 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.402260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.402260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.402313 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.402589 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:06.402590 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:06.402631 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.402713 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.402803 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260926 22:40:06.403034 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 13022
I20260926 22:40:06.403133 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:06.403329 11695 external_mini_cluster.cc:1428] Setting key 8f262afc67acef1a09e9af1dc7907397
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:40:06.426779 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:40007
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:06.505959 13162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:06.506182 13162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:06.506222 13162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:06.507390 13162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:06.507454 13162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:06.507488 13162 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:40:06.507529 13162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:06.508888 13162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.107.254:40007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:06.509143 13162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:06.509359 13162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:06.514254 13167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:06.514240 13170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:06.514240 13168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:06.514468 13162 server_base.cc:1034] running on GCE node
I20260926 22:40:06.514657 13162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:06.514839 13162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:06.516044 13162 hybrid_clock.cc:648] HybridClock initialized: now 1790462406516028 us; error 29 us; skew 500 ppm
Sep 26 22:40:06 dist-test-slave-j8sq krb5kdc[12765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462406, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:06.518606 13162 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:40:06.519059 13162 webserver.cc:460] Webserver started at http://127.11.107.195:37199/ using document root <none> and password file <none>
I20260926 22:40:06.519274 13162 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:06.519317 13162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:06.519480 13162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:06.520474 13162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "91938f7ee889472ab5f85936b34af595"
format_stamp: "Formatted at 2026-09-26 22:40:06 on dist-test-slave-j8sq"
server_key: "f01e26e5792aea347c81fb3e23e8a04d"
server_key_iv: "506444396088fdef51f57cdb264f43d5"
server_key_version: "encryptionkey@0"
I20260926 22:40:06.520828 13162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "91938f7ee889472ab5f85936b34af595"
format_stamp: "Formatted at 2026-09-26 22:40:06 on dist-test-slave-j8sq"
server_key: "f01e26e5792aea347c81fb3e23e8a04d"
server_key_iv: "506444396088fdef51f57cdb264f43d5"
server_key_version: "encryptionkey@0"
I20260926 22:40:06.522118 13162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:40:06.522823 13177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:06.523010 13162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:06.523093 13162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "91938f7ee889472ab5f85936b34af595"
format_stamp: "Formatted at 2026-09-26 22:40:06 on dist-test-slave-j8sq"
server_key: "f01e26e5792aea347c81fb3e23e8a04d"
server_key_iv: "506444396088fdef51f57cdb264f43d5"
server_key_version: "encryptionkey@0"
I20260926 22:40:06.523164 13162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:06.535758 13162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:06.537490 13162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:06.537645 13162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:06.538206 13162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:06.538272 13162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:06.538311 13162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:06.538335 13162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 22:40:06 dist-test-slave-j8sq krb5kdc[12765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462406, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:06.545449 13162 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:39295
I20260926 22:40:06.545764 13292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:39295 every 8 connection(s)
I20260926 22:40:06.545890 13162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:40:06.547618 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.538548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:40481 (local address 127.11.107.254:40007)
0926 22:40:06.538596 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.538598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.538842 (+ 244us) server_negotiation.cc:407] Connection header received
0926 22:40:06.539266 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.539267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.539281 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.539366 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:06.539940 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.540800 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.541800 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.542050 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.542133 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.544805 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:06.544806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:06.544807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:06.544816 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:06.546562 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.546904 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.546905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.546905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.546954 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.547156 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.547156 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.547157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.547253 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:06.547301 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.547507 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.547558 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:40:06.547837 13186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.538532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:40007 (local address 127.11.107.195:40481)
0926 22:40:06.538699 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.538736 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.539168 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.539411 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.539414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.539631 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:06.539864 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.539868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.540906 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.540908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.541698 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.541700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.542142 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.542144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.542277 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.542801 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:06.542815 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:06.544694 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:06.546628 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.546632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.546633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.546838 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.547010 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.547011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.547011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.547098 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.547307 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:06.547311 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:06.547413 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.547487 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.547593 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260926 22:40:06.548273 13293 heartbeater.cc:347] Connected to a master server at 127.11.107.254:40007
I20260926 22:40:06.548352 13293 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:06.548532 13293 heartbeater.cc:511] Master 127.11.107.254:40007 requested a full tablet report, sending...
I20260926 22:40:06.548875 12809 ts_manager.cc:194] Registered new tserver with Master: 91938f7ee889472ab5f85936b34af595 (127.11.107.195:39295)
I20260926 22:40:06.549247 12809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:40481
I20260926 22:40:06.551156 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 13162
I20260926 22:40:06.551254 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:06.551451 11695 external_mini_cluster.cc:1428] Setting key da340ccf5300c01e56abd11409c28a67
I20260926 22:40:06.552217 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:40:06.554682 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.549276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:40007 (local address 127.11.107.195:49707)
0926 22:40:06.549416 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.549427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.549485 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.549627 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.549628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.549771 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:06.549823 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.549824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.550485 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.550486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.551023 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.551025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.551399 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.551401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.551473 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.551894 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:06.551905 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:06.552152 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:06.553629 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.553630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.553631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.553904 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.554127 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.554128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.554129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.554201 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.554465 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:06.554466 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:06.554498 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.554548 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.554615 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
I20260926 22:40:06.554767 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.549325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:49707 (local address 127.11.107.254:40007)
0926 22:40:06.549365 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.549367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.549471 (+ 104us) server_negotiation.cc:407] Connection header received
0926 22:40:06.549549 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.549549 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.549559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.549607 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:06.549920 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.550407 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.551099 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.551330 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.551388 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.552272 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:06.552273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:06.552274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:06.552280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:06.553549 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.553981 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.553982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.553983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.554047 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.554269 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.554270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.554270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.554400 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:06.554471 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.554572 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.554646 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 22:40:06 dist-test-slave-j8sq krb5kdc[12765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462406, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 22:40:06 dist-test-slave-j8sq krb5kdc[12765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462406, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:06.574443 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.568105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.11.107.254:40007)
0926 22:40:06.568265 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.568267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.568274 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:06.568293 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.568293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.568301 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.568347 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:06.568873 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.569326 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.570050 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.570218 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.570267 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.572188 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:06.572189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:06.572189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:06.572195 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:06.573527 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.573793 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.573793 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.573794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.573897 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.574068 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.574069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.574069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.574156 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:06.574227 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.574306 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.574357 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260926 22:40:06.575892 12809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:06.577432 12809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:06.583597 13307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.579465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:36615 (local address 127.0.0.1:47818)
0926 22:40:06.579778 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.579821 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.579873 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.580333 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.580334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.580335 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:06.580364 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.580364 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.581626 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.581627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.582603 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.582605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.583305 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.583306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.583384 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.583397 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.583464 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.583513 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":79,"threads_started":1}
I20260926 22:40:06.583727 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.579586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.11.107.194:36645)
0926 22:40:06.580013 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.580018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.580029 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:40:06.580068 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.580070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.580086 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.580168 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:06.580315 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.581202 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.582601 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.583356 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.583443 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.583494 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.583591 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.583634 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":87,"threads_started":1}
I20260926 22:40:06.583917 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.579615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47818 (local address 127.11.107.193:36615)
0926 22:40:06.580013 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.580018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.580029 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:40:06.580068 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.580070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.580086 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.580179 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:06.580443 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.581567 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.582672 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.583217 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.583308 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.583364 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.583772 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.583825 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":105,"threads_started":1}
I20260926 22:40:06.583693 13311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.579467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:36645 (local address 127.0.0.1:56760)
0926 22:40:06.579808 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.579825 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.579882 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.580170 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.580173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.580175 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:06.580254 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.580255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.581283 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.581284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.582522 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.582525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.583440 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.583441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.583510 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.583513 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.583562 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.583592 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":82,"threads_started":1}
I20260926 22:40:06.584129 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.579691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:39295 (local address 127.0.0.1:58730)
0926 22:40:06.579971 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.579980 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.580030 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.580246 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.580247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.580248 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:06.580286 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.580287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.581550 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.581551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.582733 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.582735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.583658 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.583659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.583908 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.583912 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.583961 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.584031 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":36,"threads_started":1}
I20260926 22:40:06.584343 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.579802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.11.107.195:39295)
0926 22:40:06.579982 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.579987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.580035 (+ 48us) server_negotiation.cc:407] Connection header received
0926 22:40:06.580105 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.580107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.580121 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.580186 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:06.580360 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.581469 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.582808 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.583509 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.583587 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.583631 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.584199 (+ 568us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.584241 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260926 22:40:06.584750 12947 tablet_service.cc:1467] Processing CreateTablet for tablet 60e416aadab7426291f8b84929a1954f (DEFAULT_TABLE table=test-table [id=4eaa898f80d4484e811c0f448c2c7457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:06.584741 13074 tablet_service.cc:1467] Processing CreateTablet for tablet 60e416aadab7426291f8b84929a1954f (DEFAULT_TABLE table=test-table [id=4eaa898f80d4484e811c0f448c2c7457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:06.585021 12947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e416aadab7426291f8b84929a1954f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:06.585103 13074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e416aadab7426291f8b84929a1954f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:06.587843 13313 tablet_bootstrap.cc:492] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874: Bootstrap starting.
I20260926 22:40:06.587941 13314 tablet_bootstrap.cc:492] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e: Bootstrap starting.
I20260926 22:40:06.588346 13211 tablet_service.cc:1467] Processing CreateTablet for tablet 60e416aadab7426291f8b84929a1954f (DEFAULT_TABLE table=test-table [id=4eaa898f80d4484e811c0f448c2c7457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:06.588500 13313 tablet_bootstrap.cc:654] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:06.588498 13314 tablet_bootstrap.cc:654] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:06.588657 13211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e416aadab7426291f8b84929a1954f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:06.588852 13314 log.cc:824] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:06.588852 13313 log.cc:824] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:06.589579 13314 tablet_bootstrap.cc:492] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e: No bootstrap required, opened a new log
I20260926 22:40:06.589649 13314 ts_tablet_manager.cc:1397] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:06.589808 13313 tablet_bootstrap.cc:492] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874: No bootstrap required, opened a new log
I20260926 22:40:06.589882 13313 ts_tablet_manager.cc:1397] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:06.591565 13317 tablet_bootstrap.cc:492] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595: Bootstrap starting.
I20260926 22:40:06.591673 13314 raft_consensus.cc:348] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.591846 13314 raft_consensus.cc:374] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:06.591887 13314 raft_consensus.cc:729] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d26d9be0d7e41239388dda163b1523e, State: Initialized, Role: FOLLOWER
I20260926 22:40:06.591866 13313 raft_consensus.cc:348] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.592022 13313 raft_consensus.cc:374] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:06.592005 13314 consensus_queue.cc:260] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.592053 13313 raft_consensus.cc:729] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd74331259c74083ba60c3798a062874, State: Initialized, Role: FOLLOWER
I20260926 22:40:06.592182 13313 consensus_queue.cc:260] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.592252 13314 ts_tablet_manager.cc:1428] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:40:06.592355 13153 heartbeater.cc:503] Master 127.11.107.254:40007 was elected leader, sending a full tablet report...
I20260926 22:40:06.592571 13317 tablet_bootstrap.cc:654] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:06.592729 13313 ts_tablet_manager.cc:1428] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 22:40:06.592787 13013 heartbeater.cc:503] Master 127.11.107.254:40007 was elected leader, sending a full tablet report...
I20260926 22:40:06.592901 13317 log.cc:824] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:06.593595 13317 tablet_bootstrap.cc:492] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595: No bootstrap required, opened a new log
I20260926 22:40:06.593685 13317 ts_tablet_manager.cc:1397] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:40:06.595026 13317 raft_consensus.cc:348] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.595162 13317 raft_consensus.cc:374] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:06.595207 13317 raft_consensus.cc:729] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91938f7ee889472ab5f85936b34af595, State: Initialized, Role: FOLLOWER
I20260926 22:40:06.595350 13317 consensus_queue.cc:260] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.595607 13317 ts_tablet_manager.cc:1428] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:40:06.595690 13293 heartbeater.cc:503] Master 127.11.107.254:40007 was elected leader, sending a full tablet report...
W20260926 22:40:06.644073 13154 tablet.cc:2369] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:40:06.732852 13319 raft_consensus.cc:482] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:06.732980 13319 raft_consensus.cc:504] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.733412 13319 leader_election.cc:288] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d26d9be0d7e41239388dda163b1523e (127.11.107.194:36645), 91938f7ee889472ab5f85936b34af595 (127.11.107.195:39295)
I20260926 22:40:06.737346 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.733672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:39435 (local address 127.11.107.194:36645)
0926 22:40:06.733719 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.733721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.733747 (+ 26us) server_negotiation.cc:407] Connection header received
0926 22:40:06.733774 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.733774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.733783 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.733829 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:06.734101 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.734840 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.736419 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.736980 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.737061 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.737069 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.737240 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.737276 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:40:06.737346 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.733777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:38345 (local address 127.11.107.195:39295)
0926 22:40:06.733830 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.733832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.733866 (+ 34us) server_negotiation.cc:407] Connection header received
0926 22:40:06.733897 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.733898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.733906 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.733960 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:06.734099 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.734854 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.736419 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.736984 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.737061 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.737068 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.737240 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.737276 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 22:40:06.737367 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.733577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:36645 (local address 127.11.107.193:39435)
0926 22:40:06.733652 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.733666 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.733714 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.733911 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.733911 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.733913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:06.734010 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.734013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.734928 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.734929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.736255 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.736258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.737068 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.737069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.737174 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.737177 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.737227 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.737259 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":55}
I20260926 22:40:06.737376 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.733598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:39295 (local address 127.11.107.193:38345)
0926 22:40:06.733741 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.733753 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.733809 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.733971 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.733971 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.733973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:06.734029 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.734031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.734928 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.734929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.736256 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.736258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.737069 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.737069 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.737148 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.737151 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.737210 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.737259 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":95,"thread_start_us":78,"threads_started":1}
I20260926 22:40:06.737735 13107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e416aadab7426291f8b84929a1954f" candidate_uuid: "bd74331259c74083ba60c3798a062874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d26d9be0d7e41239388dda163b1523e" is_pre_election: true
I20260926 22:40:06.737821 13245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e416aadab7426291f8b84929a1954f" candidate_uuid: "bd74331259c74083ba60c3798a062874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91938f7ee889472ab5f85936b34af595" is_pre_election: true
I20260926 22:40:06.737933 13107 raft_consensus.cc:2454] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd74331259c74083ba60c3798a062874 in term 0.
I20260926 22:40:06.738018 13245 raft_consensus.cc:2454] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd74331259c74083ba60c3798a062874 in term 0.
I20260926 22:40:06.738135 12898 leader_election.cc:302] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d26d9be0d7e41239388dda163b1523e, bd74331259c74083ba60c3798a062874; no voters:
I20260926 22:40:06.738263 13319 raft_consensus.cc:2781] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:06.738330 13319 raft_consensus.cc:482] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:06.738364 13319 raft_consensus.cc:3037] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:06.739071 13319 raft_consensus.cc:504] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.739225 13319 leader_election.cc:288] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [CANDIDATE]: Term 1 election: Requested vote from peers 9d26d9be0d7e41239388dda163b1523e (127.11.107.194:36645), 91938f7ee889472ab5f85936b34af595 (127.11.107.195:39295)
I20260926 22:40:06.739385 13245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e416aadab7426291f8b84929a1954f" candidate_uuid: "bd74331259c74083ba60c3798a062874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91938f7ee889472ab5f85936b34af595"
I20260926 22:40:06.739372 13107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e416aadab7426291f8b84929a1954f" candidate_uuid: "bd74331259c74083ba60c3798a062874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d26d9be0d7e41239388dda163b1523e"
I20260926 22:40:06.739462 13245 raft_consensus.cc:3037] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:06.739485 13107 raft_consensus.cc:3037] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:06.740159 13245 raft_consensus.cc:2454] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd74331259c74083ba60c3798a062874 in term 1.
I20260926 22:40:06.740182 13107 raft_consensus.cc:2454] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd74331259c74083ba60c3798a062874 in term 1.
I20260926 22:40:06.740332 12901 leader_election.cc:302] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91938f7ee889472ab5f85936b34af595, bd74331259c74083ba60c3798a062874; no voters:
I20260926 22:40:06.740409 13319 raft_consensus.cc:2781] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:06.740567 13319 raft_consensus.cc:686] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [term 1 LEADER]: Becoming Leader. State: Replica: bd74331259c74083ba60c3798a062874, State: Running, Role: LEADER
I20260926 22:40:06.740681 13319 consensus_queue.cc:237] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.741554 12809 catalog_manager.cc:5495] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd74331259c74083ba60c3798a062874 (127.11.107.193). New cstate: current_term: 1 leader_uuid: "bd74331259c74083ba60c3798a062874" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } health_report { overall_health: UNKNOWN } } }
W20260926 22:40:06.754144 13014 tablet.cc:2369] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:40:06.764184 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.761965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.11.107.193:36615)
0926 22:40:06.762022 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.762025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.762062 (+ 37us) server_negotiation.cc:407] Connection header received
0926 22:40:06.762155 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.762156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.762171 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.762229 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:06.762353 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.762809 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.763517 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.763693 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.763740 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.763904 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:06.763989 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:06.764043 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.764088 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.764105 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:40:06.765888 13245 raft_consensus.cc:1261] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595 [term 1 FOLLOWER]: Refusing update from remote peer bd74331259c74083ba60c3798a062874: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:06.765900 13107 raft_consensus.cc:1261] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e [term 1 FOLLOWER]: Refusing update from remote peer bd74331259c74083ba60c3798a062874: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:06.766284 13323 consensus_queue.cc:1035] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:06.766446 13319 consensus_queue.cc:1035] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:06.769304 13327 mvcc.cc:205] Tried to move back new op lower bound from 7333734018110435328 to 7333734018010656768. Current Snapshot: MvccSnapshot[applied={T|T < 7333734018110435328}]
I20260926 22:40:06.772028 12809 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:06.772105 12809 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:06.772862 12809 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728: Sending DeleteTablet for 3 replicas of tablet 60e416aadab7426291f8b84929a1954f
I20260926 22:40:06.773146 13074 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e416aadab7426291f8b84929a1954f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:06 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:56760
I20260926 22:40:06.773164 12947 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e416aadab7426291f8b84929a1954f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:06 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:47818
I20260926 22:40:06.773353 13211 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e416aadab7426291f8b84929a1954f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:06 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:58730
I20260926 22:40:06.773535 13338 tablet_replica.cc:330] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595: stopping tablet replica
I20260926 22:40:06.773559 13337 tablet_replica.cc:330] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874: stopping tablet replica
I20260926 22:40:06.773576 12809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 22:40:06.773648 13337 raft_consensus.cc:2229] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:06.773671 13338 raft_consensus.cc:2229] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 22:40:06.773700 12809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:06.773738 13337 raft_consensus.cc:2258] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:06.773775 13338 raft_consensus.cc:2258] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:06.774446 13337 ts_tablet_manager.cc:1905] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:06.774513 13338 ts_tablet_manager.cc:1905] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:06.775027 12947 tablet_service.cc:1467] Processing CreateTablet for tablet 8371a2395d59437ca732422c0d739494 (DEFAULT_TABLE table=test-table [id=ae68897424f64f9e9d76979a4f24f36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:06.775153 12947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8371a2395d59437ca732422c0d739494. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:06.775225 13074 tablet_service.cc:1467] Processing CreateTablet for tablet 8371a2395d59437ca732422c0d739494 (DEFAULT_TABLE table=test-table [id=ae68897424f64f9e9d76979a4f24f36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:06.775339 13074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8371a2395d59437ca732422c0d739494. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:06.776448 13313 tablet_bootstrap.cc:492] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874: Bootstrap starting.
I20260926 22:40:06.776527 13337 ts_tablet_manager.cc:1918] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:06.776597 13337 log.cc:1196] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/wal/wals/60e416aadab7426291f8b84929a1954f
I20260926 22:40:06.776721 13314 tablet_bootstrap.cc:492] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e: Bootstrap starting.
I20260926 22:40:06.776885 13313 tablet_bootstrap.cc:654] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:06.776912 13337 ts_tablet_manager.cc:1939] T 60e416aadab7426291f8b84929a1954f P bd74331259c74083ba60c3798a062874: Deleting consensus metadata
I20260926 22:40:06.777105 13314 tablet_bootstrap.cc:654] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:06.777100 13338 ts_tablet_manager.cc:1918] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:06.777199 13338 log.cc:1196] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/60e416aadab7426291f8b84929a1954f
I20260926 22:40:06.777374 13211 tablet_service.cc:1467] Processing CreateTablet for tablet 8371a2395d59437ca732422c0d739494 (DEFAULT_TABLE table=test-table [id=ae68897424f64f9e9d76979a4f24f36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:06.777463 13211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8371a2395d59437ca732422c0d739494. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:06.777453 13338 ts_tablet_manager.cc:1939] T 60e416aadab7426291f8b84929a1954f P 91938f7ee889472ab5f85936b34af595: Deleting consensus metadata
I20260926 22:40:06.777750 12797 catalog_manager.cc:4841] TS bd74331259c74083ba60c3798a062874 (127.11.107.193:36615): tablet 60e416aadab7426291f8b84929a1954f (table test-table [id=4eaa898f80d4484e811c0f448c2c7457]) successfully deleted
I20260926 22:40:06.777917 12797 catalog_manager.cc:4841] TS 91938f7ee889472ab5f85936b34af595 (127.11.107.195:39295): tablet 60e416aadab7426291f8b84929a1954f (table test-table [id=4eaa898f80d4484e811c0f448c2c7457]) successfully deleted
I20260926 22:40:06.778628 13317 tablet_bootstrap.cc:492] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595: Bootstrap starting.
I20260926 22:40:06.779021 13317 tablet_bootstrap.cc:654] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:06.779656 13317 tablet_bootstrap.cc:492] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595: No bootstrap required, opened a new log
I20260926 22:40:06.779723 13317 ts_tablet_manager.cc:1397] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:06.779881 13317 raft_consensus.cc:348] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.779966 13317 raft_consensus.cc:374] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:06.779995 13317 raft_consensus.cc:729] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91938f7ee889472ab5f85936b34af595, State: Initialized, Role: FOLLOWER
I20260926 22:40:06.780045 13317 consensus_queue.cc:260] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.780144 13317 ts_tablet_manager.cc:1428] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:06.781060 13313 tablet_bootstrap.cc:492] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874: No bootstrap required, opened a new log
I20260926 22:40:06.781124 13313 ts_tablet_manager.cc:1397] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260926 22:40:06.781275 13313 raft_consensus.cc:348] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.781347 13313 raft_consensus.cc:374] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:06.781378 13313 raft_consensus.cc:729] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd74331259c74083ba60c3798a062874, State: Initialized, Role: FOLLOWER
I20260926 22:40:06.781425 13313 consensus_queue.cc:260] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.781512 13313 ts_tablet_manager.cc:1428] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:06.781944 13336 tablet_replica.cc:330] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e: stopping tablet replica
I20260926 22:40:06.782063 13336 raft_consensus.cc:2229] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:06.782196 13336 raft_consensus.cc:2258] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:06.782611 13336 ts_tablet_manager.cc:1905] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:06.782763 13314 tablet_bootstrap.cc:492] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e: No bootstrap required, opened a new log
I20260926 22:40:06.782810 13314 ts_tablet_manager.cc:1397] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 22:40:06.782955 13314 raft_consensus.cc:348] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.783028 13314 raft_consensus.cc:374] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:06.783056 13314 raft_consensus.cc:729] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d26d9be0d7e41239388dda163b1523e, State: Initialized, Role: FOLLOWER
I20260926 22:40:06.783098 13314 consensus_queue.cc:260] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.783178 13314 ts_tablet_manager.cc:1428] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:06.784261 13336 ts_tablet_manager.cc:1918] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:06.784334 13336 log.cc:1196] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/60e416aadab7426291f8b84929a1954f
I20260926 22:40:06.784572 13336 ts_tablet_manager.cc:1939] T 60e416aadab7426291f8b84929a1954f P 9d26d9be0d7e41239388dda163b1523e: Deleting consensus metadata
I20260926 22:40:06.785068 12794 catalog_manager.cc:4841] TS 9d26d9be0d7e41239388dda163b1523e (127.11.107.194:36645): tablet 60e416aadab7426291f8b84929a1954f (table test-table [id=4eaa898f80d4484e811c0f448c2c7457]) successfully deleted
W20260926 22:40:06.796483 13294 tablet.cc:2369] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:40:06.896129 13321 raft_consensus.cc:482] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:06.896224 13321 raft_consensus.cc:504] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.896621 13321 leader_election.cc:288] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd74331259c74083ba60c3798a062874 (127.11.107.193:36615), 9d26d9be0d7e41239388dda163b1523e (127.11.107.194:36645)
I20260926 22:40:06.900609 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.897018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:35723 (local address 127.11.107.194:36645)
0926 22:40:06.897084 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.897086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.897093 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:06.897117 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.897118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.897126 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.897178 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:06.897374 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.898165 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.899693 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.900244 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.900326 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.900335 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.900500 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.900540 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 22:40:06.900646 13186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.896813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:36645 (local address 127.11.107.195:35723)
0926 22:40:06.896881 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.896896 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.896939 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.897208 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.897208 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.897210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:06.897297 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.897299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.898238 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.898239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.899576 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.899579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.900332 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.900333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.900411 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.900418 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.900476 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.900531 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":70}
I20260926 22:40:06.900655 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.896958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:37287 (local address 127.11.107.193:36615)
0926 22:40:06.897054 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.897056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.897066 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:40:06.897096 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.897097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.897109 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.897169 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:06.897377 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.898146 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.899693 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.900242 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.900326 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.900335 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.900498 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.900585 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 22:40:06.900880 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.896829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:36615 (local address 127.11.107.195:37287)
0926 22:40:06.896924 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.896935 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.896991 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.897207 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.897208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.897210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:06.897303 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.897304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.898238 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.898239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.899576 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.899579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.900350 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.900351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.900411 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.900418 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.900476 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.900531 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":63,"thread_start_us":57,"threads_started":1}
I20260926 22:40:06.901014 13107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8371a2395d59437ca732422c0d739494" candidate_uuid: "91938f7ee889472ab5f85936b34af595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d26d9be0d7e41239388dda163b1523e" is_pre_election: true
I20260926 22:40:06.901118 13107 raft_consensus.cc:2454] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91938f7ee889472ab5f85936b34af595 in term 0.
I20260926 22:40:06.901158 12967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8371a2395d59437ca732422c0d739494" candidate_uuid: "91938f7ee889472ab5f85936b34af595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd74331259c74083ba60c3798a062874" is_pre_election: true
I20260926 22:40:06.901310 12967 raft_consensus.cc:2454] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91938f7ee889472ab5f85936b34af595 in term 0.
I20260926 22:40:06.901288 13178 leader_election.cc:302] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91938f7ee889472ab5f85936b34af595, 9d26d9be0d7e41239388dda163b1523e; no voters:
I20260926 22:40:06.901443 13321 raft_consensus.cc:2781] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:06.901490 13321 raft_consensus.cc:482] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:06.901530 13321 raft_consensus.cc:3037] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:06.902304 13321 raft_consensus.cc:504] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.902457 13321 leader_election.cc:288] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [CANDIDATE]: Term 1 election: Requested vote from peers bd74331259c74083ba60c3798a062874 (127.11.107.193:36615), 9d26d9be0d7e41239388dda163b1523e (127.11.107.194:36645)
I20260926 22:40:06.902598 12967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8371a2395d59437ca732422c0d739494" candidate_uuid: "91938f7ee889472ab5f85936b34af595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd74331259c74083ba60c3798a062874"
I20260926 22:40:06.902601 13107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8371a2395d59437ca732422c0d739494" candidate_uuid: "91938f7ee889472ab5f85936b34af595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d26d9be0d7e41239388dda163b1523e"
I20260926 22:40:06.902673 13107 raft_consensus.cc:3037] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:06.902689 12967 raft_consensus.cc:3037] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:06.903339 13107 raft_consensus.cc:2454] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91938f7ee889472ab5f85936b34af595 in term 1.
I20260926 22:40:06.903355 12967 raft_consensus.cc:2454] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91938f7ee889472ab5f85936b34af595 in term 1.
I20260926 22:40:06.903512 13181 leader_election.cc:302] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91938f7ee889472ab5f85936b34af595, bd74331259c74083ba60c3798a062874; no voters:
I20260926 22:40:06.903607 13321 raft_consensus.cc:2781] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:06.903740 13321 raft_consensus.cc:686] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [term 1 LEADER]: Becoming Leader. State: Replica: 91938f7ee889472ab5f85936b34af595, State: Running, Role: LEADER
I20260926 22:40:06.903822 13321 consensus_queue.cc:237] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.904549 12809 catalog_manager.cc:5495] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91938f7ee889472ab5f85936b34af595 (127.11.107.195). New cstate: current_term: 1 leader_uuid: "91938f7ee889472ab5f85936b34af595" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } health_report { overall_health: HEALTHY } } }
Sep 26 22:40:06 dist-test-slave-j8sq krb5kdc[12765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462406, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:06.949421 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.942645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49594 (local address 127.11.107.254:40007)
0926 22:40:06.942776 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.942778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.942831 (+ 53us) server_negotiation.cc:407] Connection header received
0926 22:40:06.942874 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.942875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.942883 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.942938 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:06.943438 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.944021 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.944671 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.944858 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.944917 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.946940 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:06.946942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:06.946942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:06.946948 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:06.948418 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.948811 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.948812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.948812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.948862 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:06.949041 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:06.949042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:06.949042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:06.949142 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:06.949190 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.949304 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.949356 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 22:40:06.949585 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.942581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:40007 (local address 127.0.0.1:49594)
0926 22:40:06.942748 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.942758 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.942825 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.942954 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.942955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.943262 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:06.943340 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.943343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.944089 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.944090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.944588 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.944589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.944930 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.944930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.944970 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.945457 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:06.945461 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:06.946855 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:06.948485 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.948491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.948493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.948739 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.948921 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:06.948921 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:06.948922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:06.948979 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:06.949195 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:06.949201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:06.949239 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.949290 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.949417 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
I20260926 22:40:06.951635 12809 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:49594:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 22:40:06.951846 12809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:06.953364 13211 tablet_service.cc:1467] Processing CreateTablet for tablet b296a600ced84cd09ec0cfde463084c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd71eccc3ac845c2ab97478b8e84f203]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:40:06.953373 13074 tablet_service.cc:1467] Processing CreateTablet for tablet b296a600ced84cd09ec0cfde463084c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd71eccc3ac845c2ab97478b8e84f203]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:40:06.953521 13211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b296a600ced84cd09ec0cfde463084c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:06.953547 13074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b296a600ced84cd09ec0cfde463084c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:06.953986 12947 tablet_service.cc:1467] Processing CreateTablet for tablet b296a600ced84cd09ec0cfde463084c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd71eccc3ac845c2ab97478b8e84f203]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:40:06.954142 12947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b296a600ced84cd09ec0cfde463084c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:06.955039 13317 tablet_bootstrap.cc:492] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595: Bootstrap starting.
I20260926 22:40:06.955459 13317 tablet_bootstrap.cc:654] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:06.955590 13313 tablet_bootstrap.cc:492] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874: Bootstrap starting.
I20260926 22:40:06.955884 13314 tablet_bootstrap.cc:492] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e: Bootstrap starting.
I20260926 22:40:06.955909 13313 tablet_bootstrap.cc:654] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:06.956293 13314 tablet_bootstrap.cc:654] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:06.956542 13313 tablet_bootstrap.cc:492] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874: No bootstrap required, opened a new log
I20260926 22:40:06.956598 13313 ts_tablet_manager.cc:1397] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:06.956713 13313 raft_consensus.cc:348] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.956787 13313 raft_consensus.cc:374] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:06.956818 13313 raft_consensus.cc:729] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd74331259c74083ba60c3798a062874, State: Initialized, Role: FOLLOWER
I20260926 22:40:06.956857 13317 tablet_bootstrap.cc:492] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595: No bootstrap required, opened a new log
I20260926 22:40:06.956900 13317 ts_tablet_manager.cc:1397] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:06.956867 13313 consensus_queue.cc:260] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.956966 13314 tablet_bootstrap.cc:492] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e: No bootstrap required, opened a new log
I20260926 22:40:06.956986 13313 ts_tablet_manager.cc:1428] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:06.957006 13314 ts_tablet_manager.cc:1397] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:06.957033 13333 tablet_replica.cc:439] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } } }
I20260926 22:40:06.957067 13317 raft_consensus.cc:348] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.957122 13317 raft_consensus.cc:374] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:06.957118 13333 tablet_replica.cc:442] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:06.957144 13317 raft_consensus.cc:729] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91938f7ee889472ab5f85936b34af595, State: Initialized, Role: FOLLOWER
I20260926 22:40:06.957145 13314 raft_consensus.cc:348] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.957208 13314 raft_consensus.cc:374] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:06.957223 13314 raft_consensus.cc:729] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d26d9be0d7e41239388dda163b1523e, State: Initialized, Role: FOLLOWER
I20260926 22:40:06.957196 13317 consensus_queue.cc:260] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.957257 13314 consensus_queue.cc:260] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.957304 13317 ts_tablet_manager.cc:1428] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:06.957312 13314 ts_tablet_manager.cc:1428] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:06.957427 13321 tablet_replica.cc:439] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } } }
I20260926 22:40:06.957487 13321 tablet_replica.cc:442] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:06.957602 13318 tablet_replica.cc:439] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } } }
I20260926 22:40:06.957690 13318 tablet_replica.cc:442] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:06.970522 13318 raft_consensus.cc:482] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:06.970585 13318 raft_consensus.cc:504] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.970844 13318 leader_election.cc:288] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd74331259c74083ba60c3798a062874 (127.11.107.193:36615), 91938f7ee889472ab5f85936b34af595 (127.11.107.195:39295)
I20260926 22:40:06.973779 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.971061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:54915 (local address 127.11.107.193:36615)
0926 22:40:06.971110 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.971113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.971196 (+ 83us) server_negotiation.cc:407] Connection header received
0926 22:40:06.971270 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.971272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.971281 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.971349 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:06.971517 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.971980 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.973028 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.973401 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.973494 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.973505 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.973676 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.973711 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 22:40:06.973811 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.971171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:39295 (local address 127.11.107.194:33769)
0926 22:40:06.971261 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.971271 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.971325 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.971467 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.971468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.971470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:06.971545 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.971547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.972146 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.972147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.972933 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.972935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.973528 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.973529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.973605 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.973609 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.973674 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.973717 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":34,"threads_started":1}
I20260926 22:40:06.973815 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.970996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:36615 (local address 127.11.107.194:54915)
0926 22:40:06.971138 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.971151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.971210 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.971367 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.971367 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.971369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:06.971437 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.971439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.972086 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.972087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.972931 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.972933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.973519 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.973520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.973589 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.973593 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.973667 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.973710 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":24,"thread_start_us":68,"threads_started":1}
I20260926 22:40:06.973821 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.971231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:33769 (local address 127.11.107.195:39295)
0926 22:40:06.971272 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.971274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.971310 (+ 36us) server_negotiation.cc:407] Connection header received
0926 22:40:06.971383 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.971385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.971393 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.971463 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:06.971616 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.972064 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.973047 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.973450 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.973529 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.973537 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.973702 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.973735 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:40:06.974133 12967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b296a600ced84cd09ec0cfde463084c6" candidate_uuid: "9d26d9be0d7e41239388dda163b1523e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd74331259c74083ba60c3798a062874" is_pre_election: true
I20260926 22:40:06.974135 13245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b296a600ced84cd09ec0cfde463084c6" candidate_uuid: "9d26d9be0d7e41239388dda163b1523e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91938f7ee889472ab5f85936b34af595" is_pre_election: true
I20260926 22:40:06.974229 12967 raft_consensus.cc:2454] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d26d9be0d7e41239388dda163b1523e in term 0.
I20260926 22:40:06.974229 13245 raft_consensus.cc:2454] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d26d9be0d7e41239388dda163b1523e in term 0.
I20260926 22:40:06.974436 13041 leader_election.cc:302] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d26d9be0d7e41239388dda163b1523e, bd74331259c74083ba60c3798a062874; no voters:
I20260926 22:40:06.974592 13318 raft_consensus.cc:2781] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:06.974647 13318 raft_consensus.cc:482] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:06.974677 13318 raft_consensus.cc:3037] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:06.975337 13318 raft_consensus.cc:504] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.975476 13318 leader_election.cc:288] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [CANDIDATE]: Term 1 election: Requested vote from peers bd74331259c74083ba60c3798a062874 (127.11.107.193:36615), 91938f7ee889472ab5f85936b34af595 (127.11.107.195:39295)
I20260926 22:40:06.975612 12967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b296a600ced84cd09ec0cfde463084c6" candidate_uuid: "9d26d9be0d7e41239388dda163b1523e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd74331259c74083ba60c3798a062874"
I20260926 22:40:06.975696 12967 raft_consensus.cc:3037] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:06.975654 13245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b296a600ced84cd09ec0cfde463084c6" candidate_uuid: "9d26d9be0d7e41239388dda163b1523e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91938f7ee889472ab5f85936b34af595"
I20260926 22:40:06.975757 13245 raft_consensus.cc:3037] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:06.976464 12967 raft_consensus.cc:2454] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d26d9be0d7e41239388dda163b1523e in term 1.
I20260926 22:40:06.976642 13041 leader_election.cc:302] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d26d9be0d7e41239388dda163b1523e, bd74331259c74083ba60c3798a062874; no voters:
I20260926 22:40:06.976730 13318 raft_consensus.cc:2781] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:06.976840 13318 raft_consensus.cc:686] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [term 1 LEADER]: Becoming Leader. State: Replica: 9d26d9be0d7e41239388dda163b1523e, State: Running, Role: LEADER
I20260926 22:40:06.976912 13245 raft_consensus.cc:2454] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d26d9be0d7e41239388dda163b1523e in term 1.
I20260926 22:40:06.976904 13318 consensus_queue.cc:237] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } }
I20260926 22:40:06.977243 13356 tablet_replica.cc:439] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e: TxnStatusTablet state changed. Reason: New leader 9d26d9be0d7e41239388dda163b1523e. Latest consensus state: current_term: 1 leader_uuid: "9d26d9be0d7e41239388dda163b1523e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } } }
I20260926 22:40:06.977347 13356 tablet_replica.cc:442] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:40:06.977520 13358 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 22:40:06.977624 13358 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 22:40:06.977608 12810 catalog_manager.cc:5495] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d26d9be0d7e41239388dda163b1523e (127.11.107.194). New cstate: current_term: 1 leader_uuid: "9d26d9be0d7e41239388dda163b1523e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } health_report { overall_health: UNKNOWN } } }
I20260926 22:40:06.987340 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.984611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:36645 (local address 127.0.0.1:56774)
0926 22:40:06.984692 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.984704 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.984754 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.984932 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.984933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.984934 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:40:06.984974 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.984976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.985885 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.985886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.986438 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.986439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.986904 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.986905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.986946 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.986959 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:40:06.987212 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:40:06.987217 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:40:06.987254 (+ 37us) client_negotiation.cc:240] Negotiation successful
0926 22:40:06.987285 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 22:40:06.987350 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.984686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.11.107.194:36645)
0926 22:40:06.984724 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.984726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.984739 (+ 13us) server_negotiation.cc:407] Connection header received
0926 22:40:06.984829 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.984830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.984858 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.984926 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:06.985039 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.985770 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.986523 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.986823 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.986892 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.987038 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:06.987148 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:06.987223 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.987260 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.987281 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:40:06.988474 13245 raft_consensus.cc:1261] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595 [term 1 FOLLOWER]: Refusing update from remote peer 9d26d9be0d7e41239388dda163b1523e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:06.988597 12967 raft_consensus.cc:1261] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874 [term 1 FOLLOWER]: Refusing update from remote peer 9d26d9be0d7e41239388dda163b1523e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:06.988781 13356 consensus_queue.cc:1035] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [LEADER]: Connected to new peer: Peer: permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:06.988914 13318 consensus_queue.cc:1035] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:06.989600 13333 tablet_replica.cc:439] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874: TxnStatusTablet state changed. Reason: New leader 9d26d9be0d7e41239388dda163b1523e. Latest consensus state: current_term: 1 leader_uuid: "9d26d9be0d7e41239388dda163b1523e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } } }
I20260926 22:40:06.989712 13333 tablet_replica.cc:442] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:06.989807 13321 tablet_replica.cc:439] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595: TxnStatusTablet state changed. Reason: New leader 9d26d9be0d7e41239388dda163b1523e. Latest consensus state: current_term: 1 leader_uuid: "9d26d9be0d7e41239388dda163b1523e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } } }
I20260926 22:40:06.989924 13321 tablet_replica.cc:442] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:06.990625 13356 tablet_replica.cc:439] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d26d9be0d7e41239388dda163b1523e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } } }
I20260926 22:40:06.990706 13356 tablet_replica.cc:442] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:40:06.991408 13326 mvcc.cc:205] Tried to move back new op lower bound from 7333734019022761984 to 7333734018978009088. Current Snapshot: MvccSnapshot[applied={T|T < 7333734019022761984}]
I20260926 22:40:06.991493 13333 tablet_replica.cc:439] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d26d9be0d7e41239388dda163b1523e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } } }
I20260926 22:40:06.991577 13333 tablet_replica.cc:442] T b296a600ced84cd09ec0cfde463084c6 P bd74331259c74083ba60c3798a062874: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:06.992561 13318 tablet_replica.cc:439] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d26d9be0d7e41239388dda163b1523e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } } }
I20260926 22:40:06.992640 13318 tablet_replica.cc:442] T b296a600ced84cd09ec0cfde463084c6 P 9d26d9be0d7e41239388dda163b1523e: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:40:06.993259 13328 mvcc.cc:205] Tried to move back new op lower bound from 7333734019022761984 to 7333734018978009088. Current Snapshot: MvccSnapshot[applied={T|T < 7333734019022761984}]
I20260926 22:40:06.993355 13321 tablet_replica.cc:439] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d26d9be0d7e41239388dda163b1523e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 } } peers { permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 } } peers { permanent_uuid: "91938f7ee889472ab5f85936b34af595" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39295 } } }
I20260926 22:40:06.993423 13321 tablet_replica.cc:442] T b296a600ced84cd09ec0cfde463084c6 P 91938f7ee889472ab5f85936b34af595: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:06.995121 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.992373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.11.107.195:39295)
0926 22:40:06.992465 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.992467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.992476 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:06.992511 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.992511 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.992531 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.992596 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:06.992757 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.993413 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.994324 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.994543 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.994622 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.994792 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:06.994902 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:06.994986 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:06.995026 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 22:40:06.995044 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 22:40:07.000101 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.997529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:36645 (local address 127.11.107.195:54885)
0926 22:40:06.997605 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:06.997616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:06.997658 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:06.997791 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:06.997792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:06.997793 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:40:06.997879 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.997881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.998757 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.998758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.999301 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:06.999302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.999766 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.999767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:06.999812 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.999825 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:40:06.999987 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:40:06.999989 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:40:07.000023 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 22:40:07.000054 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 22:40:07.000113 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:06.997594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:54885 (local address 127.11.107.194:36645)
0926 22:40:06.997631 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:06.997633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:06.997642 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:06.997712 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:06.997713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:06.997723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:06.997788 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:06.997929 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.998681 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.999380 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:06.999695 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:06.999764 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:06.999899 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:06.999930 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:06.999982 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:07.000026 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 22:40:07.000045 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:40:07.002506 12967 raft_consensus.cc:1261] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874 [term 1 FOLLOWER]: Refusing update from remote peer 91938f7ee889472ab5f85936b34af595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:07.002547 13107 raft_consensus.cc:1261] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e [term 1 FOLLOWER]: Refusing update from remote peer 91938f7ee889472ab5f85936b34af595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:07.002748 13343 consensus_queue.cc:1035] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d26d9be0d7e41239388dda163b1523e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 36645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:07.002910 13321 consensus_queue.cc:1035] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd74331259c74083ba60c3798a062874" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:07.011557 13370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:07.008530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:39295 (local address 127.11.107.194:39461)
0926 22:40:07.008678 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:07.008688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:07.008748 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:07.008868 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:07.008869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:07.008870 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:40:07.008925 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:07.008926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:07.009775 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:07.009775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:07.010700 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:07.010703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:07.011182 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:07.011183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:07.011229 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:07.011243 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:40:07.011412 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:40:07.011414 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:40:07.011448 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 22:40:07.011481 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260926 22:40:07.011571 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:07.008552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:39461 (local address 127.11.107.195:39295)
0926 22:40:07.008592 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:07.008594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:07.008760 (+ 166us) server_negotiation.cc:407] Connection header received
0926 22:40:07.008800 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:07.008800 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:07.008811 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:07.008867 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:07.008962 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:07.009703 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:07.010772 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:07.011084 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:07.011155 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:07.011319 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:07.011353 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:07.011407 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:07.011451 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 22:40:07.011470 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:40:07.020337 12809 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:07.020462 12809 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:07.021051 12809 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f0371c5c489407ebf3efad44792f728: Sending DeleteTablet for 3 replicas of tablet 8371a2395d59437ca732422c0d739494
I20260926 22:40:07.021347 13074 tablet_service.cc:1514] Processing DeleteTablet for tablet 8371a2395d59437ca732422c0d739494 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:07 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:56760
I20260926 22:40:07.021405 13211 tablet_service.cc:1514] Processing DeleteTablet for tablet 8371a2395d59437ca732422c0d739494 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:07 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:58730
I20260926 22:40:07.021507 13336 tablet_replica.cc:330] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e: stopping tablet replica
I20260926 22:40:07.021591 13336 raft_consensus.cc:2229] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:07.021631 13336 raft_consensus.cc:2258] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:07.021711 13338 tablet_replica.cc:330] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595: stopping tablet replica
I20260926 22:40:07.021760 13338 raft_consensus.cc:2229] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:07.021858 13338 raft_consensus.cc:2258] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:07.022133 12947 tablet_service.cc:1514] Processing DeleteTablet for tablet 8371a2395d59437ca732422c0d739494 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:07 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:47818
I20260926 22:40:07.022218 13337 tablet_replica.cc:330] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874: stopping tablet replica
I20260926 22:40:07.022274 13337 raft_consensus.cc:2229] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:07.022274 13338 ts_tablet_manager.cc:1905] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:07.022323 13337 raft_consensus.cc:2258] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:07.022612 13336 ts_tablet_manager.cc:1905] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:07.024132 13336 ts_tablet_manager.cc:1918] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:07.024168 13338 ts_tablet_manager.cc:1918] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:07.024192 13336 log.cc:1196] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/8371a2395d59437ca732422c0d739494
I20260926 22:40:07.024220 13338 log.cc:1196] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/8371a2395d59437ca732422c0d739494
I20260926 22:40:07.024412 13336 ts_tablet_manager.cc:1939] T 8371a2395d59437ca732422c0d739494 P 9d26d9be0d7e41239388dda163b1523e: Deleting consensus metadata
I20260926 22:40:07.024448 13338 ts_tablet_manager.cc:1939] T 8371a2395d59437ca732422c0d739494 P 91938f7ee889472ab5f85936b34af595: Deleting consensus metadata
I20260926 22:40:07.024729 12794 catalog_manager.cc:4841] TS 9d26d9be0d7e41239388dda163b1523e (127.11.107.194:36645): tablet 8371a2395d59437ca732422c0d739494 (table test-table [id=ae68897424f64f9e9d76979a4f24f36d]) successfully deleted
I20260926 22:40:07.024943 13337 ts_tablet_manager.cc:1905] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:07.026522 13337 ts_tablet_manager.cc:1918] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:07.026585 13337 log.cc:1196] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462400949426-11695-0/minicluster-data/ts-0/wal/wals/8371a2395d59437ca732422c0d739494
I20260926 22:40:07.026796 13337 ts_tablet_manager.cc:1939] T 8371a2395d59437ca732422c0d739494 P bd74331259c74083ba60c3798a062874: Deleting consensus metadata
I20260926 22:40:07.027179 12797 catalog_manager.cc:4841] TS bd74331259c74083ba60c3798a062874 (127.11.107.193:36615): tablet 8371a2395d59437ca732422c0d739494 (table test-table [id=ae68897424f64f9e9d76979a4f24f36d]) successfully deleted
I20260926 22:40:07.027252 12797 catalog_manager.cc:4841] TS 91938f7ee889472ab5f85936b34af595 (127.11.107.195:39295): tablet 8371a2395d59437ca732422c0d739494 (table test-table [id=ae68897424f64f9e9d76979a4f24f36d]) successfully deleted
Sep 26 22:40:07 dist-test-slave-j8sq krb5kdc[12765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462406, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.107.193@KRBTEST.COM
I20260926 22:40:07.038877 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:07.027920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.11.107.193:36615)
0926 22:40:07.027977 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:07.027979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:07.028059 (+ 80us) server_negotiation.cc:407] Connection header received
0926 22:40:07.028120 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:07.028121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:07.028131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:07.028221 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:07.029019 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:07.029828 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:07.030972 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:07.031270 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:07.031349 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:07.035295 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:07.035306 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:07.035306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:07.035315 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:07.037354 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:07.037780 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:07.037785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:07.037785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:07.037930 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:07.038154 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:07.038154 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:07.038155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:07.038343 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:07.038487 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:07.038578 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 22:40:07.038669 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 22:40:07.039093 13007 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47828
I20260926 22:40:07.047757 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:07.040071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49608 (local address 127.11.107.254:40007)
0926 22:40:07.040120 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:07.040123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:07.040245 (+ 122us) server_negotiation.cc:407] Connection header received
0926 22:40:07.040295 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:07.040296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:07.040307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:07.040372 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:07.040983 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:07.041799 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:07.042845 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:07.043160 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:07.043235 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:07.044707 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:07.044709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:07.044709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:07.044717 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:07.046474 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:07.046903 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:07.046904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:07.046904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:07.046961 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:07.047176 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:07.047177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:07.047177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:07.047325 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:07.047436 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:07.047564 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 22:40:07.047648 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 22:40:07.048329 12809 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49608
I20260926 22:40:07.049202 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 12882
I20260926 22:40:07.057654 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 13022
I20260926 22:40:07.063656 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 13162
I20260926 22:40:07.068742 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 12778
2026-09-26T22:40:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2293 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:07 dist-test-slave-j8sq krb5kdc[13379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:07 dist-test-slave-j8sq krb5kdc[13379](info): set up 2 sockets
Sep 26 22:40:07 dist-test-slave-j8sq krb5kdc[13379](info): commencing operation
krb5kdc: starting...
W20260926 22:40:08.106539 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:08 dist-test-slave-j8sq krb5kdc[13379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:40:08.219969 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:38267
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:38267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:08.299369 13392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:08.299568 13392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:08.299589 13392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:08.300729 13392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:08.300767 13392 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 22:40:08.300781 13392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:08.300818 13392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:08.300846 13392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:08.300861 13392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:08.302273 13392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:38267
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:38267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:08.302510 13392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:08.302711 13392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:08.307459 13400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:08.307477 13397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:08.307458 13398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:08.307504 13392 server_base.cc:1034] running on GCE node
I20260926 22:40:08.308024 13392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:08.308329 13392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:08.309489 13392 hybrid_clock.cc:648] HybridClock initialized: now 1790462408309485 us; error 35 us; skew 500 ppm
Sep 26 22:40:08 dist-test-slave-j8sq krb5kdc[13379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462408, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:08.311908 13392 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:40:08.312278 13392 webserver.cc:460] Webserver started at http://127.11.107.254:36905/ using document root <none> and password file <none>
I20260926 22:40:08.312445 13392 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:08.312482 13392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:08.312567 13392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:08.313508 13392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "e6307e1b4f054e04a85b9426decc21a6"
format_stamp: "Formatted at 2026-09-26 22:40:08 on dist-test-slave-j8sq"
server_key: "52de430ab4aeeaaab28744f45acb1979"
server_key_iv: "7e1798f16c7e51e03a84bea555a910d9"
server_key_version: "encryptionkey@0"
I20260926 22:40:08.313795 13392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "e6307e1b4f054e04a85b9426decc21a6"
format_stamp: "Formatted at 2026-09-26 22:40:08 on dist-test-slave-j8sq"
server_key: "52de430ab4aeeaaab28744f45acb1979"
server_key_iv: "7e1798f16c7e51e03a84bea555a910d9"
server_key_version: "encryptionkey@0"
I20260926 22:40:08.315071 13392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:08.315745 13407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:08.315893 13392 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:08.315971 13392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "e6307e1b4f054e04a85b9426decc21a6"
format_stamp: "Formatted at 2026-09-26 22:40:08 on dist-test-slave-j8sq"
server_key: "52de430ab4aeeaaab28744f45acb1979"
server_key_iv: "7e1798f16c7e51e03a84bea555a910d9"
server_key_version: "encryptionkey@0"
I20260926 22:40:08.316025 13392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:08.331256 13392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:08.332988 13392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:08.333096 13392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:08.337484 13392 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:38267
I20260926 22:40:08.337572 13469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:38267 every 8 connection(s)
I20260926 22:40:08.337978 13392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:08.338677 13470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:08.340831 13470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6: Bootstrap starting.
I20260926 22:40:08.341415 13470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:08.341725 13470 log.cc:824] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:08.342401 13470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6: No bootstrap required, opened a new log
I20260926 22:40:08.344393 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 13392
I20260926 22:40:08.344462 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:08.344466 13470 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6307e1b4f054e04a85b9426decc21a6" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38267 } }
I20260926 22:40:08.344635 11695 external_mini_cluster.cc:1428] Setting key 78f469209e84c08098ad6ede70e13353
I20260926 22:40:08.344655 13470 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:08.344707 13470 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6307e1b4f054e04a85b9426decc21a6, State: Initialized, Role: FOLLOWER
I20260926 22:40:08.344866 13470 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6307e1b4f054e04a85b9426decc21a6" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38267 } }
I20260926 22:40:08.344955 13470 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:08.344990 13470 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:08.345041 13470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:08.345721 13470 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6307e1b4f054e04a85b9426decc21a6" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38267 } }
I20260926 22:40:08.345832 13470 leader_election.cc:302] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6307e1b4f054e04a85b9426decc21a6; no voters:
I20260926 22:40:08.346024 13470 leader_election.cc:288] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:08.346077 13475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:08.346215 13475 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [term 1 LEADER]: Becoming Leader. State: Replica: e6307e1b4f054e04a85b9426decc21a6, State: Running, Role: LEADER
I20260926 22:40:08.346283 13470 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:08.346329 13475 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6307e1b4f054e04a85b9426decc21a6" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38267 } }
I20260926 22:40:08.346709 13475 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6307e1b4f054e04a85b9426decc21a6. Latest consensus state: current_term: 1 leader_uuid: "e6307e1b4f054e04a85b9426decc21a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6307e1b4f054e04a85b9426decc21a6" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38267 } } }
I20260926 22:40:08.346773 13475 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:08.346657 13477 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6307e1b4f054e04a85b9426decc21a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6307e1b4f054e04a85b9426decc21a6" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38267 } } }
I20260926 22:40:08.346920 13477 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:08.347102 13480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:08.347710 13480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:08.349427 13480 catalog_manager.cc:1347] Generated new cluster ID: 0cd2912b651648b5b0be5ccf4f04f269
I20260926 22:40:08.349490 13480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:40:08 dist-test-slave-j8sq krb5kdc[13379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:08.356917 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.345122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.11.107.254:38267)
0926 22:40:08.345313 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.345326 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.345351 (+ 25us) server_negotiation.cc:407] Connection header received
0926 22:40:08.345442 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.345447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.345488 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.345602 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:08.346476 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.348000 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.348977 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.349553 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.349672 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.352490 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:08.352495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:08.352495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:08.352508 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:08.355208 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.355640 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.355643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.355644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.355763 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.355979 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.355979 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.355980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.356280 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:08.356426 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.356558 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.356691 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":115,"threads_started":1}
I20260926 22:40:08.361253 13480 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:08.361975 13480 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:08.367300 13480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6: Generated new TSK 0
I20260926 22:40:08.367475 13480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:40:08.394610 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38267
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:08.473102 13496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:08.473330 13496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:08.473368 13496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:08.474476 13496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:08.474545 13496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:08.474576 13496 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 22:40:08.474628 13496 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:40:08.474658 13496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:08.476064 13496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.107.254:38267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:08.476325 13496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:08.476547 13496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:08.481228 13504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:08.481237 13502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:08.481227 13501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:08.481398 13496 server_base.cc:1034] running on GCE node
I20260926 22:40:08.481608 13496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:08.481918 13496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:08.483089 13496 hybrid_clock.cc:648] HybridClock initialized: now 1790462408483082 us; error 29 us; skew 500 ppm
Sep 26 22:40:08 dist-test-slave-j8sq krb5kdc[13379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462408, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:08.485417 13496 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:40:08.485884 13496 webserver.cc:460] Webserver started at http://127.11.107.193:37113/ using document root <none> and password file <none>
I20260926 22:40:08.486122 13496 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:08.486192 13496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:08.486338 13496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:08.487335 13496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db"
format_stamp: "Formatted at 2026-09-26 22:40:08 on dist-test-slave-j8sq"
server_key: "40034695c67c362ee6d710c0587a57c1"
server_key_iv: "33db636a5a1ff5fd97250981920e9dc9"
server_key_version: "encryptionkey@0"
I20260926 22:40:08.487690 13496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db"
format_stamp: "Formatted at 2026-09-26 22:40:08 on dist-test-slave-j8sq"
server_key: "40034695c67c362ee6d710c0587a57c1"
server_key_iv: "33db636a5a1ff5fd97250981920e9dc9"
server_key_version: "encryptionkey@0"
I20260926 22:40:08.488988 13496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:08.489653 13511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:08.489826 13496 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:08.489945 13496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db"
format_stamp: "Formatted at 2026-09-26 22:40:08 on dist-test-slave-j8sq"
server_key: "40034695c67c362ee6d710c0587a57c1"
server_key_iv: "33db636a5a1ff5fd97250981920e9dc9"
server_key_version: "encryptionkey@0"
I20260926 22:40:08.490029 13496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:08.513934 13496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:08.515744 13496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:08.515908 13496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:08.516405 13496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:08.516470 13496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:08.516517 13496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:08.516546 13496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 22:40:08 dist-test-slave-j8sq krb5kdc[13379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462408, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:08.523561 13496 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:41121
I20260926 22:40:08.523613 13626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:41121 every 8 connection(s)
I20260926 22:40:08.523984 13496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:08.526779 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.516936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:38859 (local address 127.11.107.254:38267)
0926 22:40:08.517012 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.517015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.517188 (+ 173us) server_negotiation.cc:407] Connection header received
0926 22:40:08.517652 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.517654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.517670 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.517752 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:08.518626 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.519217 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.520721 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.521053 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.521125 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.523851 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:08.523853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:08.523853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:08.523860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:08.525337 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.525762 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.525764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.525764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.525837 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.526115 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.526116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.526117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.526240 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:08.526329 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.526605 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.526686 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 22:40:08.526978 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.516873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:38267 (local address 127.11.107.193:38859)
0926 22:40:08.517043 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:08.517082 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:08.517558 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:08.517844 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:08.517866 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:08.518230 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:08.518521 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.518527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.519312 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.519313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.520636 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.520639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.521171 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.521173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.521345 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.521919 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:08.521932 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:08.523760 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:08.525439 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.525446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.525448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.525671 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.525923 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.525924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.525925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.526030 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.526308 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:08.526314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:08.526474 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 22:40:08.526584 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 22:40:08.526743 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260926 22:40:08.527527 13627 heartbeater.cc:347] Connected to a master server at 127.11.107.254:38267
I20260926 22:40:08.527612 13627 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:08.527803 13627 heartbeater.cc:511] Master 127.11.107.254:38267 requested a full tablet report, sending...
I20260926 22:40:08.528345 13423 ts_manager.cc:194] Registered new tserver with Master: afe9b2c1ad5f42b8a79ba0a7da24b8db (127.11.107.193:41121)
I20260926 22:40:08.529143 13423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:38859
I20260926 22:40:08.529513 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 13496
I20260926 22:40:08.529626 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:08.529798 11695 external_mini_cluster.cc:1428] Setting key 6a296cbfec561c04ccfd3aea72507deb
I20260926 22:40:08.535200 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.528614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:38267 (local address 127.11.107.193:43535)
0926 22:40:08.528775 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:08.528800 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:08.528851 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:08.529621 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:08.529622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:08.529808 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:08.529905 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.529907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.530754 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.530754 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.531241 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.531243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.531732 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.531733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.531820 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.532210 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:08.532216 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:08.532601 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:08.534325 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.534326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.534327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.534482 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.534686 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.534686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.534686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.534734 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.534988 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:08.534989 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:08.535021 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 22:40:08.535069 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 22:40:08.535138 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260926 22:40:08.535465 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.528604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:43535 (local address 127.11.107.254:38267)
0926 22:40:08.529484 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.529486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.529496 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:40:08.529516 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.529517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.529526 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.529582 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:08.529974 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.530652 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.531321 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.531637 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.531725 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.532702 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:08.532703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:08.532704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:08.532709 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:08.534251 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.534557 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.534559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.534559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.534618 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.534793 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.534794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.534794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.534927 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:08.534996 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.535098 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.535182 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857}
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:40:08.553879 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38267
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:08.632015 13636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:08.632213 13636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:08.632256 13636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:08.633406 13636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:08.633447 13636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:08.633493 13636 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 22:40:08.633522 13636 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:40:08.633553 13636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:08.634964 13636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.107.254:38267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:08.635200 13636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:08.635422 13636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:08.640309 13641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:08.640355 13636 server_base.cc:1034] running on GCE node
W20260926 22:40:08.640372 13642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:08.640293 13644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:08.640619 13636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:08.640874 13636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:08.642006 13636 hybrid_clock.cc:648] HybridClock initialized: now 1790462408642001 us; error 35 us; skew 500 ppm
Sep 26 22:40:08 dist-test-slave-j8sq krb5kdc[13379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462408, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:08.644265 13636 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:40:08.644752 13636 webserver.cc:460] Webserver started at http://127.11.107.194:33441/ using document root <none> and password file <none>
I20260926 22:40:08.644979 13636 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:08.645043 13636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:08.645190 13636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:08.646219 13636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "c60952adeddc454d9fd5b7580bd840e3"
format_stamp: "Formatted at 2026-09-26 22:40:08 on dist-test-slave-j8sq"
server_key: "39b498c04a8b6667621ca4dac6cc9f4d"
server_key_iv: "c68efe51dbfd98ed7052c269c13c815e"
server_key_version: "encryptionkey@0"
I20260926 22:40:08.646682 13636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "c60952adeddc454d9fd5b7580bd840e3"
format_stamp: "Formatted at 2026-09-26 22:40:08 on dist-test-slave-j8sq"
server_key: "39b498c04a8b6667621ca4dac6cc9f4d"
server_key_iv: "c68efe51dbfd98ed7052c269c13c815e"
server_key_version: "encryptionkey@0"
I20260926 22:40:08.648093 13636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:08.648769 13651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:08.648948 13636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:08.649015 13636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "c60952adeddc454d9fd5b7580bd840e3"
format_stamp: "Formatted at 2026-09-26 22:40:08 on dist-test-slave-j8sq"
server_key: "39b498c04a8b6667621ca4dac6cc9f4d"
server_key_iv: "c68efe51dbfd98ed7052c269c13c815e"
server_key_version: "encryptionkey@0"
I20260926 22:40:08.649094 13636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:08.666213 13636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:08.667878 13636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:08.668030 13636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:08.668537 13636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:08.668576 13636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:08.668596 13636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:08.668639 13636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 22:40:08 dist-test-slave-j8sq krb5kdc[13379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462408, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:08.675979 13636 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:35437
I20260926 22:40:08.676373 13636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:40:08.676789 13766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:35437 every 8 connection(s)
I20260926 22:40:08.677980 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 13636
I20260926 22:40:08.678079 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:08.678260 11695 external_mini_cluster.cc:1428] Setting key 139eb2ea60a14c4d48368ef0ece6b567
I20260926 22:40:08.678725 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.668916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:53281 (local address 127.11.107.254:38267)
0926 22:40:08.668977 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.668979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.669162 (+ 183us) server_negotiation.cc:407] Connection header received
0926 22:40:08.669633 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.669635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.669647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.669689 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:08.670388 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.670873 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.671839 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.672129 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.672219 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.675574 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:08.675576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:08.675577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:08.675583 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:08.677113 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.677600 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.677601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.677602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.677652 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.677975 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.677976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.677976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.678130 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:08.678229 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.678578 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.678659 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 22:40:08.678861 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.668896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:38267 (local address 127.11.107.194:53281)
0926 22:40:08.669042 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:08.669078 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:08.669535 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:08.669760 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:08.669763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:08.670116 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:08.670298 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.670302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.670962 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.670963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.671748 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.671750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.672227 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.672228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.672382 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.673055 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:08.673066 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:08.675464 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:08.677194 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.677198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.677200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.677535 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.677744 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.677745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.677746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.677906 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.678220 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:08.678226 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:08.678384 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 22:40:08.678488 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 22:40:08.678654 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260926 22:40:08.679272 13767 heartbeater.cc:347] Connected to a master server at 127.11.107.254:38267
I20260926 22:40:08.679346 13767 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:08.679514 13767 heartbeater.cc:511] Master 127.11.107.254:38267 requested a full tablet report, sending...
I20260926 22:40:08.679808 13423 ts_manager.cc:194] Registered new tserver with Master: c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194:35437)
I20260926 22:40:08.680295 13423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:53281
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
I20260926 22:40:08.686436 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.680680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:36651 (local address 127.11.107.254:38267)
0926 22:40:08.680740 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.680743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.680934 (+ 191us) server_negotiation.cc:407] Connection header received
0926 22:40:08.680980 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.680981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.680992 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.681049 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:08.681420 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.682170 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.682817 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.683017 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.683099 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.683980 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:08.683981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:08.683982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:08.683988 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:08.685300 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.685648 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.685649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.685649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.685714 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.685901 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.685902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.685903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.686043 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:08.686122 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.686274 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.686346 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:40:08.686448 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.680721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:38267 (local address 127.11.107.194:36651)
0926 22:40:08.680849 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:08.680862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:08.680926 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:08.681052 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:08.681053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:08.681272 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:08.681348 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.681350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.682254 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.682254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.682730 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.682731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.683102 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.683103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.683181 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.683591 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:08.683596 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:08.683878 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:08.685401 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.685402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.685402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.685569 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.685778 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.685778 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.685779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.685835 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.686113 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:08.686114 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:08.686157 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 22:40:08.686251 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 22:40:08.686347 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:40:08.704487 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38267
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:08.784358 13776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:08.784569 13776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:08.784610 13776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:08.785749 13776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:08.785812 13776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:08.785851 13776 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 22:40:08.785923 13776 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:40:08.785964 13776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:08.787375 13776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.107.254:38267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:08.787650 13776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:08.787863 13776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:08.792961 13782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:08.792946 13784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:08.792946 13781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:08.793180 13776 server_base.cc:1034] running on GCE node
I20260926 22:40:08.793336 13776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:08.793515 13776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:08.794651 13776 hybrid_clock.cc:648] HybridClock initialized: now 1790462408794640 us; error 28 us; skew 500 ppm
Sep 26 22:40:08 dist-test-slave-j8sq krb5kdc[13379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462408, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:08.796902 13776 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:40:08.797300 13776 webserver.cc:460] Webserver started at http://127.11.107.195:37337/ using document root <none> and password file <none>
I20260926 22:40:08.797513 13776 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:08.797577 13776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:08.797727 13776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:08.798810 13776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "a30d6c22589c42c78e206a5206977cbd"
format_stamp: "Formatted at 2026-09-26 22:40:08 on dist-test-slave-j8sq"
server_key: "934d15677d12aadc65cae2e5284914d6"
server_key_iv: "6941f323bf03e0c586233c113b5d4b7d"
server_key_version: "encryptionkey@0"
I20260926 22:40:08.799165 13776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "a30d6c22589c42c78e206a5206977cbd"
format_stamp: "Formatted at 2026-09-26 22:40:08 on dist-test-slave-j8sq"
server_key: "934d15677d12aadc65cae2e5284914d6"
server_key_iv: "6941f323bf03e0c586233c113b5d4b7d"
server_key_version: "encryptionkey@0"
I20260926 22:40:08.800436 13776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:08.801092 13791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:08.801249 13776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:08.801373 13776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "a30d6c22589c42c78e206a5206977cbd"
format_stamp: "Formatted at 2026-09-26 22:40:08 on dist-test-slave-j8sq"
server_key: "934d15677d12aadc65cae2e5284914d6"
server_key_iv: "6941f323bf03e0c586233c113b5d4b7d"
server_key_version: "encryptionkey@0"
I20260926 22:40:08.801455 13776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:08.814669 13776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:08.816327 13776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:08.816475 13776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:08.816938 13776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:08.816998 13776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:08.817054 13776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:08.817082 13776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 22:40:08 dist-test-slave-j8sq krb5kdc[13379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462408, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:08.824093 13776 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:43679
I20260926 22:40:08.824172 13906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:43679 every 8 connection(s)
I20260926 22:40:08.824479 13776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:40:08.826766 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.817447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:59481 (local address 127.11.107.254:38267)
0926 22:40:08.817516 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.817518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.817720 (+ 202us) server_negotiation.cc:407] Connection header received
0926 22:40:08.818356 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.818358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.818372 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.818441 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:08.819253 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.820103 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.821633 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.821957 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.822042 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.824254 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:08.824256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:08.824256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:08.824264 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:08.825670 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.826066 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.826066 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.826067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.826116 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.826319 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.826319 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.826319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.826408 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:08.826453 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.826650 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.826706 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 22:40:08.826972 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.817432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:38267 (local address 127.11.107.195:59481)
0926 22:40:08.817587 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:08.817623 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:08.818260 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:08.818534 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:08.818538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:08.818864 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:08.819129 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.819136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.820206 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.820206 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.821540 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.821543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.822051 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.822052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.822190 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.822717 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:08.822733 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:08.824145 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:08.825737 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.825742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.825744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.826002 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.826188 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.826188 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.826189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.826279 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.826456 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:08.826460 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:08.826559 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 22:40:08.826637 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 22:40:08.826748 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
I20260926 22:40:08.827451 13907 heartbeater.cc:347] Connected to a master server at 127.11.107.254:38267
I20260926 22:40:08.827525 13907 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:08.827718 13907 heartbeater.cc:511] Master 127.11.107.254:38267 requested a full tablet report, sending...
I20260926 22:40:08.828080 13423 ts_manager.cc:194] Registered new tserver with Master: a30d6c22589c42c78e206a5206977cbd (127.11.107.195:43679)
I20260926 22:40:08.828519 13423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:59481
I20260926 22:40:08.828972 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 13776
I20260926 22:40:08.829079 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:08.829262 11695 external_mini_cluster.cc:1428] Setting key b9673f4d573880f64fe0c8cf02633efc
I20260926 22:40:08.830060 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:40:08.833942 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.828542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:35659 (local address 127.11.107.254:38267)
0926 22:40:08.828581 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.828583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.828638 (+ 55us) server_negotiation.cc:407] Connection header received
0926 22:40:08.828710 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.828711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.828721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.828791 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:08.829131 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.829655 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.830414 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.830619 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.830691 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.831499 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:08.831500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:08.831501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:08.831507 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:08.832843 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.833180 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.833181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.833182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.833236 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.833430 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.833431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.833431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.833533 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:08.833606 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.833758 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.833830 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:40:08.833943 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.828470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:38267 (local address 127.11.107.195:35659)
0926 22:40:08.828584 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:08.828595 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:08.828652 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:08.828794 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:08.828795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:08.828984 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:08.829038 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.829040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.829753 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.829754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.830340 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.830342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.830692 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.830693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.830766 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.831164 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:08.831169 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:08.831412 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:08.832940 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.832941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.832941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.833108 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.833308 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.833309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.833309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.833368 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.833598 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:08.833599 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:08.833642 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 22:40:08.833736 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 22:40:08.833831 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
Sep 26 22:40:08 dist-test-slave-j8sq krb5kdc[13379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 22:40:08 dist-test-slave-j8sq krb5kdc[13379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:08.852141 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.845501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.11.107.254:38267)
0926 22:40:08.845557 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.845559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.845581 (+ 22us) server_negotiation.cc:407] Connection header received
0926 22:40:08.845640 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.845641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.845649 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.845700 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:08.846318 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.846821 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.847560 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.847783 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.847849 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.850078 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:08.850079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:08.850080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:08.850086 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:08.851190 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.851475 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.851475 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.851476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.851556 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.851734 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.851734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.851735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.851828 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:08.851916 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.852023 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.852078 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:40:08.853730 13423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:08.855393 13423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:08.861313 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.857741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.11.107.193:41121)
0926 22:40:08.857947 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.857952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.857963 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:40:08.858072 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.858074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.858091 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.858164 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:08.858408 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.859105 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.860141 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.860903 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.860978 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.861028 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.861180 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.861216 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260926 22:40:08.861301 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.857537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:41121 (local address 127.0.0.1:46916)
0926 22:40:08.857700 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:08.857740 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:08.857807 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:08.858240 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:08.858244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:08.858246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:08.858347 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.858349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.859213 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.859214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.860049 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.860052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.860995 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.860996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.861070 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.861082 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 22:40:08.861155 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 22:40:08.861220 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":23,"thread_start_us":52,"threads_started":1}
I20260926 22:40:08.861436 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.857551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35437 (local address 127.0.0.1:59180)
0926 22:40:08.857925 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:08.857937 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:08.857995 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:08.858254 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:08.858255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:08.858256 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:08.858330 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.858332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.859178 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.859179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.860401 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.860404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.861127 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.861128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.861196 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.861199 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:08.861244 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 22:40:08.861273 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":33,"thread_start_us":66,"threads_started":1}
I20260926 22:40:08.861694 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.857765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59180 (local address 127.11.107.194:35437)
0926 22:40:08.857963 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.857968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.857980 (+ 12us) server_negotiation.cc:407] Connection header received
0926 22:40:08.858093 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.858095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.858110 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.858192 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:08.858416 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.859086 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.860471 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.860971 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.861042 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.861096 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.861341 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.861590 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":98,"threads_started":1}
I20260926 22:40:08.862252 13545 tablet_service.cc:1467] Processing CreateTablet for tablet 6d128bec5f254357b1e865b679d235d3 (DEFAULT_TABLE table=test-table [id=fb85a895e7ac471f8899416774fd13dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:08.862300 13701 tablet_service.cc:1467] Processing CreateTablet for tablet 6d128bec5f254357b1e865b679d235d3 (DEFAULT_TABLE table=test-table [id=fb85a895e7ac471f8899416774fd13dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:08.862627 13701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d128bec5f254357b1e865b679d235d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:08.862627 13545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d128bec5f254357b1e865b679d235d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:08.862707 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.857537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:43679 (local address 127.0.0.1:51312)
0926 22:40:08.857905 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:08.857915 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:08.857954 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:08.858278 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:08.858279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:08.858281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:08.858343 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.858345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.860420 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.860421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.861563 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.861566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.862470 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.862471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.862545 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.862549 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:08.862598 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:40:08.862633 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":8,"thread_start_us":49,"threads_started":1}
I20260926 22:40:08.862959 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.857735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.11.107.195:43679)
0926 22:40:08.858054 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.858058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.858067 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:08.858104 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.858106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.858122 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.858209 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:08.859234 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.860338 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.861633 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.862286 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.862385 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.862438 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.862820 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.862871 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":84,"threads_started":1}
I20260926 22:40:08.863495 13841 tablet_service.cc:1467] Processing CreateTablet for tablet 6d128bec5f254357b1e865b679d235d3 (DEFAULT_TABLE table=test-table [id=fb85a895e7ac471f8899416774fd13dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:08.863737 13841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d128bec5f254357b1e865b679d235d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:08.866389 13929 tablet_bootstrap.cc:492] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Bootstrap starting.
I20260926 22:40:08.866416 13927 tablet_bootstrap.cc:492] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd: Bootstrap starting.
I20260926 22:40:08.866931 13929 tablet_bootstrap.cc:654] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:08.867238 13929 log.cc:824] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:08.867255 13927 tablet_bootstrap.cc:654] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:08.867537 13927 log.cc:824] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:08.867558 13928 tablet_bootstrap.cc:492] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3: Bootstrap starting.
I20260926 22:40:08.868144 13928 tablet_bootstrap.cc:654] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:08.868287 13927 tablet_bootstrap.cc:492] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd: No bootstrap required, opened a new log
I20260926 22:40:08.868362 13927 ts_tablet_manager.cc:1397] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:08.868476 13928 log.cc:824] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:08.869019 13929 tablet_bootstrap.cc:492] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db: No bootstrap required, opened a new log
I20260926 22:40:08.869081 13929 ts_tablet_manager.cc:1397] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 22:40:08.869997 13927 raft_consensus.cc:348] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } }
I20260926 22:40:08.870144 13927 raft_consensus.cc:374] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:08.870194 13927 raft_consensus.cc:729] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30d6c22589c42c78e206a5206977cbd, State: Initialized, Role: FOLLOWER
I20260926 22:40:08.870324 13927 consensus_queue.cc:260] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } }
I20260926 22:40:08.870610 13927 ts_tablet_manager.cc:1428] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:40:08.870620 13907 heartbeater.cc:503] Master 127.11.107.254:38267 was elected leader, sending a full tablet report...
I20260926 22:40:08.870615 13929 raft_consensus.cc:348] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } }
I20260926 22:40:08.870790 13929 raft_consensus.cc:374] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:08.870831 13929 raft_consensus.cc:729] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe9b2c1ad5f42b8a79ba0a7da24b8db, State: Initialized, Role: FOLLOWER
I20260926 22:40:08.870939 13929 consensus_queue.cc:260] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } }
I20260926 22:40:08.871189 13929 ts_tablet_manager.cc:1428] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:40:08.871227 13627 heartbeater.cc:503] Master 127.11.107.254:38267 was elected leader, sending a full tablet report...
I20260926 22:40:08.871563 13928 tablet_bootstrap.cc:492] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3: No bootstrap required, opened a new log
I20260926 22:40:08.871644 13928 ts_tablet_manager.cc:1397] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 22:40:08.873293 13928 raft_consensus.cc:348] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } }
I20260926 22:40:08.873407 13928 raft_consensus.cc:374] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:08.873428 13928 raft_consensus.cc:729] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60952adeddc454d9fd5b7580bd840e3, State: Initialized, Role: FOLLOWER
I20260926 22:40:08.873555 13928 consensus_queue.cc:260] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } }
I20260926 22:40:08.873813 13928 ts_tablet_manager.cc:1428] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:40:08.873822 13767 heartbeater.cc:503] Master 127.11.107.254:38267 was elected leader, sending a full tablet report...
I20260926 22:40:08.922005 13935 raft_consensus.cc:482] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:08.922127 13935 raft_consensus.cc:504] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } }
I20260926 22:40:08.922500 13935 leader_election.cc:288] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afe9b2c1ad5f42b8a79ba0a7da24b8db (127.11.107.193:41121), a30d6c22589c42c78e206a5206977cbd (127.11.107.195:43679)
I20260926 22:40:08.926281 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.922761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:55443 (local address 127.11.107.193:41121)
0926 22:40:08.922813 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.922815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.922828 (+ 13us) server_negotiation.cc:407] Connection header received
0926 22:40:08.922890 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.922891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.922898 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.922950 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:08.923110 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.923871 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.925342 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.925928 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.926003 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.926011 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.926180 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.926214 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 22:40:08.926281 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.922794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:60151 (local address 127.11.107.195:43679)
0926 22:40:08.922839 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.922841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.923024 (+ 183us) server_negotiation.cc:407] Connection header received
0926 22:40:08.923059 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.923060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.923086 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.923136 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:08.923289 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.923963 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.925342 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.925927 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.926003 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.926011 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.926175 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.926214 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:40:08.926287 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.922694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:41121 (local address 127.11.107.194:55443)
0926 22:40:08.922771 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:08.922784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:08.922828 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:08.922959 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:08.922960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:08.922962 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:08.923034 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.923036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.923949 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.923950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.925248 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.925251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.926008 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.926009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.926099 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.926102 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:08.926154 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:40:08.926196 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":75}
I20260926 22:40:08.926295 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.922696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:43679 (local address 127.11.107.194:60151)
0926 22:40:08.922952 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:08.922960 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:08.923009 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:08.923165 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:08.923166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:08.923167 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:08.923216 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.923217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.924037 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.924037 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.925248 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.925251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.926008 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.926009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.926085 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.926089 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:08.926162 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 22:40:08.926196 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":125,"thread_start_us":109,"threads_started":1}
I20260926 22:40:08.926633 13861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d128bec5f254357b1e865b679d235d3" candidate_uuid: "c60952adeddc454d9fd5b7580bd840e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30d6c22589c42c78e206a5206977cbd" is_pre_election: true
I20260926 22:40:08.926646 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d128bec5f254357b1e865b679d235d3" candidate_uuid: "c60952adeddc454d9fd5b7580bd840e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" is_pre_election: true
I20260926 22:40:08.926784 13861 raft_consensus.cc:2454] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c60952adeddc454d9fd5b7580bd840e3 in term 0.
I20260926 22:40:08.926785 13581 raft_consensus.cc:2454] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c60952adeddc454d9fd5b7580bd840e3 in term 0.
W20260926 22:40:08.926913 13768 tablet.cc:2369] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:40:08.927004 13654 leader_election.cc:302] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a30d6c22589c42c78e206a5206977cbd, c60952adeddc454d9fd5b7580bd840e3; no voters:
I20260926 22:40:08.927150 13935 raft_consensus.cc:2781] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:08.927235 13935 raft_consensus.cc:482] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:08.927258 13935 raft_consensus.cc:3037] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:08.928193 13935 raft_consensus.cc:504] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } }
I20260926 22:40:08.928334 13935 leader_election.cc:288] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [CANDIDATE]: Term 1 election: Requested vote from peers afe9b2c1ad5f42b8a79ba0a7da24b8db (127.11.107.193:41121), a30d6c22589c42c78e206a5206977cbd (127.11.107.195:43679)
I20260926 22:40:08.928491 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d128bec5f254357b1e865b679d235d3" candidate_uuid: "c60952adeddc454d9fd5b7580bd840e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db"
I20260926 22:40:08.928488 13861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d128bec5f254357b1e865b679d235d3" candidate_uuid: "c60952adeddc454d9fd5b7580bd840e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30d6c22589c42c78e206a5206977cbd"
I20260926 22:40:08.928578 13581 raft_consensus.cc:3037] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:08.928581 13861 raft_consensus.cc:3037] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:08.929221 13581 raft_consensus.cc:2454] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c60952adeddc454d9fd5b7580bd840e3 in term 1.
I20260926 22:40:08.929289 13861 raft_consensus.cc:2454] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c60952adeddc454d9fd5b7580bd840e3 in term 1.
I20260926 22:40:08.929482 13654 leader_election.cc:302] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a30d6c22589c42c78e206a5206977cbd, c60952adeddc454d9fd5b7580bd840e3; no voters:
I20260926 22:40:08.929569 13935 raft_consensus.cc:2781] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:08.929728 13935 raft_consensus.cc:686] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [term 1 LEADER]: Becoming Leader. State: Replica: c60952adeddc454d9fd5b7580bd840e3, State: Running, Role: LEADER
I20260926 22:40:08.929817 13935 consensus_queue.cc:237] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } }
I20260926 22:40:08.930672 13424 catalog_manager.cc:5495] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194). New cstate: current_term: 1 leader_uuid: "c60952adeddc454d9fd5b7580bd840e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } health_report { overall_health: HEALTHY } } }
Sep 26 22:40:08 dist-test-slave-j8sq krb5kdc[13379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462408, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:08.977075 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.970557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.11.107.254:38267)
0926 22:40:08.970652 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:08.970654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:08.970680 (+ 26us) server_negotiation.cc:407] Connection header received
0926 22:40:08.970748 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:08.970748 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:08.970756 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:08.970805 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:08.971222 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.971697 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.972358 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.972524 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.972570 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.974545 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:08.974546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:08.974546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:08.974553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:08.975889 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.976260 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.976261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.976261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.976315 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:08.976502 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:08.976503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:08.976503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:08.976605 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:08.976693 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:08.976853 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 22:40:08.976931 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 22:40:08.977166 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:08.970445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:38267 (local address 127.0.0.1:37522)
0926 22:40:08.970628 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:08.970637 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:08.970695 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:08.970822 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:08.970823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:08.971108 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:08.971150 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.971152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.971769 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.971770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.972294 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:08.972295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:08.972583 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:08.972584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:08.972620 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:08.973081 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:08.973085 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:08.974457 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:08.975986 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.975994 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.975996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.976188 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.976379 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:08.976379 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:08.976380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:08.976434 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:08.976686 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:08.976693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:08.976736 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 22:40:08.976833 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 22:40:08.977019 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":77,"thread_start_us":85,"threads_started":1}
I20260926 22:40:08.978950 13424 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:37522:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 22:40:08.979166 13424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:08.980502 13841 tablet_service.cc:1467] Processing CreateTablet for tablet b45e2398547f4658bf3379d1fdc0e02c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9e4278dce1c4d66914df3bfe8ba506f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:40:08.980520 13545 tablet_service.cc:1467] Processing CreateTablet for tablet b45e2398547f4658bf3379d1fdc0e02c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9e4278dce1c4d66914df3bfe8ba506f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:40:08.980520 13701 tablet_service.cc:1467] Processing CreateTablet for tablet b45e2398547f4658bf3379d1fdc0e02c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9e4278dce1c4d66914df3bfe8ba506f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:40:08.980669 13701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b45e2398547f4658bf3379d1fdc0e02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:08.980670 13545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b45e2398547f4658bf3379d1fdc0e02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:08.980672 13841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b45e2398547f4658bf3379d1fdc0e02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:08.981828 13927 tablet_bootstrap.cc:492] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd: Bootstrap starting.
I20260926 22:40:08.982149 13929 tablet_bootstrap.cc:492] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db: Bootstrap starting.
I20260926 22:40:08.982455 13927 tablet_bootstrap.cc:654] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:08.982873 13929 tablet_bootstrap.cc:654] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:08.983223 13927 tablet_bootstrap.cc:492] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd: No bootstrap required, opened a new log
I20260926 22:40:08.983243 13928 tablet_bootstrap.cc:492] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3: Bootstrap starting.
I20260926 22:40:08.983275 13927 ts_tablet_manager.cc:1397] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:08.983428 13927 raft_consensus.cc:348] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:08.983521 13927 raft_consensus.cc:374] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:08.983508 13929 tablet_bootstrap.cc:492] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db: No bootstrap required, opened a new log
I20260926 22:40:08.983556 13927 raft_consensus.cc:729] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30d6c22589c42c78e206a5206977cbd, State: Initialized, Role: FOLLOWER
I20260926 22:40:08.983572 13929 ts_tablet_manager.cc:1397] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:08.983613 13927 consensus_queue.cc:260] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:08.983644 13928 tablet_bootstrap.cc:654] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:08.983717 13927 ts_tablet_manager.cc:1428] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:08.983721 13929 raft_consensus.cc:348] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:08.983804 13929 raft_consensus.cc:374] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:08.983831 13929 raft_consensus.cc:729] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe9b2c1ad5f42b8a79ba0a7da24b8db, State: Initialized, Role: FOLLOWER
I20260926 22:40:08.983882 13929 consensus_queue.cc:260] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:08.983878 13933 tablet_replica.cc:439] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } }
I20260926 22:40:08.983974 13929 ts_tablet_manager.cc:1428] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:08.983968 13933 tablet_replica.cc:442] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:08.984045 13934 tablet_replica.cc:439] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } }
I20260926 22:40:08.984136 13934 tablet_replica.cc:442] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:08.984618 13928 tablet_bootstrap.cc:492] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3: No bootstrap required, opened a new log
I20260926 22:40:08.984664 13928 ts_tablet_manager.cc:1397] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:08.984809 13928 raft_consensus.cc:348] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:08.984887 13928 raft_consensus.cc:374] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:08.984933 13928 raft_consensus.cc:729] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60952adeddc454d9fd5b7580bd840e3, State: Initialized, Role: FOLLOWER
I20260926 22:40:08.984973 13928 consensus_queue.cc:260] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:08.985082 13928 ts_tablet_manager.cc:1428] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:08.985126 13935 tablet_replica.cc:439] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } }
I20260926 22:40:08.985217 13935 tablet_replica.cc:442] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 22:40:09.025087 13628 tablet.cc:2369] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:40:09.067365 13933 raft_consensus.cc:482] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:09.067466 13933 raft_consensus.cc:504] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:09.067839 13933 leader_election.cc:288] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afe9b2c1ad5f42b8a79ba0a7da24b8db (127.11.107.193:41121), c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194:35437)
I20260926 22:40:09.070600 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.068013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35437 (local address 127.11.107.195:55805)
0926 22:40:09.068068 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:09.068080 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:09.068129 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:09.068319 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:09.068320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:09.068321 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:09.068395 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.068396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.069046 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.069048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.069775 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.069777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.070335 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.070336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.070422 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.070425 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:09.070494 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 22:40:09.070530 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 22:40:09.070600 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.068129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:55805 (local address 127.11.107.194:35437)
0926 22:40:09.068174 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:09.068176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:09.068185 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:09.068213 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:09.068213 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:09.068221 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:09.068310 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:09.068468 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.068959 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.069889 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.070254 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.070338 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.070347 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:09.070498 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 22:40:09.070533 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 22:40:09.070746 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.068129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:53539 (local address 127.11.107.193:41121)
0926 22:40:09.068181 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:09.068183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:09.068316 (+ 133us) server_negotiation.cc:407] Connection header received
0926 22:40:09.068383 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:09.068384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:09.068393 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:09.068468 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:09.068624 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.069082 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.070050 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.070392 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.070476 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.070485 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:09.070638 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 22:40:09.070675 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:40:09.070766 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.068049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:41121 (local address 127.11.107.195:53539)
0926 22:40:09.068241 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:09.068249 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:09.068325 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:09.068470 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:09.068470 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:09.068472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:09.068552 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.068553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.069169 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.069170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.069955 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.069958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.070492 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.070493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.070560 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.070568 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 22:40:09.070633 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 22:40:09.070675 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":108,"threads_started":1}
I20260926 22:40:09.070932 13717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45e2398547f4658bf3379d1fdc0e02c" candidate_uuid: "a30d6c22589c42c78e206a5206977cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60952adeddc454d9fd5b7580bd840e3" is_pre_election: true
I20260926 22:40:09.071002 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45e2398547f4658bf3379d1fdc0e02c" candidate_uuid: "a30d6c22589c42c78e206a5206977cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" is_pre_election: true
I20260926 22:40:09.071059 13717 raft_consensus.cc:2454] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a30d6c22589c42c78e206a5206977cbd in term 0.
I20260926 22:40:09.071072 13581 raft_consensus.cc:2454] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a30d6c22589c42c78e206a5206977cbd in term 0.
I20260926 22:40:09.071224 13792 leader_election.cc:302] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a30d6c22589c42c78e206a5206977cbd, afe9b2c1ad5f42b8a79ba0a7da24b8db; no voters:
I20260926 22:40:09.071363 13933 raft_consensus.cc:2781] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:09.071403 13933 raft_consensus.cc:482] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:09.071424 13933 raft_consensus.cc:3037] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:09.072073 13933 raft_consensus.cc:504] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:09.072207 13933 leader_election.cc:288] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [CANDIDATE]: Term 1 election: Requested vote from peers afe9b2c1ad5f42b8a79ba0a7da24b8db (127.11.107.193:41121), c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194:35437)
I20260926 22:40:09.072360 13717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45e2398547f4658bf3379d1fdc0e02c" candidate_uuid: "a30d6c22589c42c78e206a5206977cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60952adeddc454d9fd5b7580bd840e3"
I20260926 22:40:09.072351 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45e2398547f4658bf3379d1fdc0e02c" candidate_uuid: "a30d6c22589c42c78e206a5206977cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db"
I20260926 22:40:09.072422 13717 raft_consensus.cc:3037] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:09.072423 13581 raft_consensus.cc:3037] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:09.073143 13581 raft_consensus.cc:2454] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a30d6c22589c42c78e206a5206977cbd in term 1.
I20260926 22:40:09.073153 13717 raft_consensus.cc:2454] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a30d6c22589c42c78e206a5206977cbd in term 1.
I20260926 22:40:09.073328 13792 leader_election.cc:302] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a30d6c22589c42c78e206a5206977cbd, afe9b2c1ad5f42b8a79ba0a7da24b8db; no voters:
I20260926 22:40:09.073426 13933 raft_consensus.cc:2781] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:09.073590 13933 raft_consensus.cc:686] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [term 1 LEADER]: Becoming Leader. State: Replica: a30d6c22589c42c78e206a5206977cbd, State: Running, Role: LEADER
I20260926 22:40:09.073712 13933 consensus_queue.cc:237] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:09.074265 13949 tablet_replica.cc:439] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd: TxnStatusTablet state changed. Reason: New leader a30d6c22589c42c78e206a5206977cbd. Latest consensus state: current_term: 1 leader_uuid: "a30d6c22589c42c78e206a5206977cbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } }
I20260926 22:40:09.074348 13949 tablet_replica.cc:442] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:40:09.074508 13951 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 22:40:09.074623 13951 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 22:40:09.074646 13424 catalog_manager.cc:5495] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd reported cstate change: term changed from 0 to 1, leader changed from <none> to a30d6c22589c42c78e206a5206977cbd (127.11.107.195). New cstate: current_term: 1 leader_uuid: "a30d6c22589c42c78e206a5206977cbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } health_report { overall_health: HEALTHY } } }
W20260926 22:40:09.075045 13908 tablet.cc:2369] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:40:09.102471 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.099804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.11.107.195:43679)
0926 22:40:09.099844 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:09.099846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:09.099872 (+ 26us) server_negotiation.cc:407] Connection header received
0926 22:40:09.099967 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:09.099968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:09.099983 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:09.100039 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:09.100187 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.100629 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.101719 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.101941 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.101987 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.102187 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:09.102272 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:09.102328 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:09.102402 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 22:40:09.102417 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:40:09.102536 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.099745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:43679 (local address 127.0.0.1:51316)
0926 22:40:09.099828 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:09.099840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:09.099891 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:09.100068 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:09.100069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:09.100071 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:40:09.100124 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.100126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.100716 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.100717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.101626 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.101629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.102009 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.102010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.102082 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.102098 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:40:09.102334 (+ 236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:40:09.102342 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 22:40:09.102400 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 22:40:09.102451 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 22:40:09.104382 13717 raft_consensus.cc:1261] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Refusing update from remote peer a30d6c22589c42c78e206a5206977cbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:09.104388 13581 raft_consensus.cc:1261] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Refusing update from remote peer a30d6c22589c42c78e206a5206977cbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:09.104708 13933 consensus_queue.cc:1035] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:09.104830 13949 consensus_queue.cc:1035] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:09.105983 13935 tablet_replica.cc:439] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3: TxnStatusTablet state changed. Reason: New leader a30d6c22589c42c78e206a5206977cbd. Latest consensus state: current_term: 1 leader_uuid: "a30d6c22589c42c78e206a5206977cbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } }
I20260926 22:40:09.106002 13934 tablet_replica.cc:439] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db: TxnStatusTablet state changed. Reason: New leader a30d6c22589c42c78e206a5206977cbd. Latest consensus state: current_term: 1 leader_uuid: "a30d6c22589c42c78e206a5206977cbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } }
I20260926 22:40:09.106079 13935 tablet_replica.cc:442] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:09.106079 13934 tablet_replica.cc:442] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:09.106967 13933 tablet_replica.cc:439] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a30d6c22589c42c78e206a5206977cbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } }
I20260926 22:40:09.107043 13933 tablet_replica.cc:442] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:40:09.107189 13933 tablet_replica.cc:439] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a30d6c22589c42c78e206a5206977cbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } }
I20260926 22:40:09.107254 13933 tablet_replica.cc:442] T b45e2398547f4658bf3379d1fdc0e02c P a30d6c22589c42c78e206a5206977cbd: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:40:09.107620 13934 tablet_replica.cc:439] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a30d6c22589c42c78e206a5206977cbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } }
I20260926 22:40:09.107707 13934 tablet_replica.cc:442] T b45e2398547f4658bf3379d1fdc0e02c P afe9b2c1ad5f42b8a79ba0a7da24b8db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:09.108114 13935 tablet_replica.cc:439] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a30d6c22589c42c78e206a5206977cbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } } }
I20260926 22:40:09.108213 13935 tablet_replica.cc:442] T b45e2398547f4658bf3379d1fdc0e02c P c60952adeddc454d9fd5b7580bd840e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:09.108318 13956 mvcc.cc:205] Tried to move back new op lower bound from 7333734027689111552 to 7333734027566866432. Current Snapshot: MvccSnapshot[applied={T|T < 7333734027689111552}]
I20260926 22:40:09.111727 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.109335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.11.107.194:35437)
0926 22:40:09.109374 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:09.109376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:09.109385 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:09.109448 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:09.109449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:09.109464 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:09.109550 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:09.109677 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.110218 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.110976 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.111187 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.111261 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.111424 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:09.111518 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:09.111604 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:09.111633 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 22:40:09.111652 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:40:09.116290 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.114086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:39757 (local address 127.11.107.195:43679)
0926 22:40:09.114224 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:09.114226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:09.114232 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:40:09.114247 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:09.114247 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:09.114254 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:09.114316 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:09.114444 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.114882 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.115602 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.115809 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.115880 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.116058 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:09.116086 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:09.116142 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:09.116211 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 22:40:09.116227 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 22:40:09.116307 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.114029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:43679 (local address 127.11.107.194:39757)
0926 22:40:09.114099 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:09.114109 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:09.114148 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:09.114319 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:09.114320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:09.114321 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:40:09.114371 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.114373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.114958 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.114959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.115520 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.115522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.115878 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.115879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.115941 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.115959 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:40:09.116157 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:40:09.116159 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:40:09.116206 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:40:09.116244 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 22:40:09.118389 13861 raft_consensus.cc:1261] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Refusing update from remote peer c60952adeddc454d9fd5b7580bd840e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:09.118571 13581 raft_consensus.cc:1261] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Refusing update from remote peer c60952adeddc454d9fd5b7580bd840e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:09.118762 13937 consensus_queue.cc:1035] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:09.118855 13935 consensus_queue.cc:1035] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:09.120332 13955 mvcc.cc:205] Tried to move back new op lower bound from 7333734027747024896 to 7333734026977202176. Current Snapshot: MvccSnapshot[applied={T|T < 7333734027747024896}]
I20260926 22:40:09.120855 13954 mvcc.cc:205] Tried to move back new op lower bound from 7333734027747024896 to 7333734026977202176. Current Snapshot: MvccSnapshot[applied={T|T < 7333734027747024896}]
I20260926 22:40:09.127122 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.124705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:39113 (local address 127.11.107.194:35437)
0926 22:40:09.124755 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:09.124756 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:09.124763 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:09.124780 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:09.124781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:09.124789 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:09.124858 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:09.124995 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.125439 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.126439 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.126679 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.126760 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.126914 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:09.126944 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:09.126989 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:09.127044 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 22:40:09.127061 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 22:40:09.127158 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.124525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35437 (local address 127.11.107.195:39113)
0926 22:40:09.124592 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:09.124602 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:09.124649 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:09.124860 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:09.124861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:09.124862 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:40:09.124923 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.124924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.125525 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.125526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.126335 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.126337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.126761 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.126762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.126828 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.126842 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:40:09.126988 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:40:09.126990 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:40:09.127040 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 22:40:09.127077 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 22:40:09.136199 13423 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:09.136292 13423 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:09.136942 13423 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6: Sending DeleteTablet for 3 replicas of tablet 6d128bec5f254357b1e865b679d235d3
I20260926 22:40:09.137248 13545 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d128bec5f254357b1e865b679d235d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:09 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:46916
I20260926 22:40:09.137483 13971 tablet_replica.cc:330] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db: stopping tablet replica
I20260926 22:40:09.137652 13971 raft_consensus.cc:2229] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:09.137653 13841 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d128bec5f254357b1e865b679d235d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:09 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:51312
I20260926 22:40:09.137805 13971 raft_consensus.cc:2258] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:09.137907 13701 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d128bec5f254357b1e865b679d235d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:09 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:59180
I20260926 22:40:09.138116 13423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:09.138288 13423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:09.138542 13971 ts_tablet_manager.cc:1905] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:09.139468 13701 tablet_service.cc:1467] Processing CreateTablet for tablet 971c61fb570b43cb9038572678b5c954 (DEFAULT_TABLE table=test-table [id=d9f4e19d8d4b4534a84a7a52ebda9f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:09.139592 13701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971c61fb570b43cb9038572678b5c954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:09.140435 13971 ts_tablet_manager.cc:1918] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:09.140563 13971 log.cc:1196] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/wal/wals/6d128bec5f254357b1e865b679d235d3
I20260926 22:40:09.140892 13971 ts_tablet_manager.cc:1939] T 6d128bec5f254357b1e865b679d235d3 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Deleting consensus metadata
I20260926 22:40:09.140971 13928 tablet_bootstrap.cc:492] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3: Bootstrap starting.
I20260926 22:40:09.141407 13928 tablet_bootstrap.cc:654] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:09.141769 13841 tablet_service.cc:1467] Processing CreateTablet for tablet 971c61fb570b43cb9038572678b5c954 (DEFAULT_TABLE table=test-table [id=d9f4e19d8d4b4534a84a7a52ebda9f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:09.141898 13841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971c61fb570b43cb9038572678b5c954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:09.142822 13927 tablet_bootstrap.cc:492] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd: Bootstrap starting.
I20260926 22:40:09.143074 13928 tablet_bootstrap.cc:492] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3: No bootstrap required, opened a new log
I20260926 22:40:09.143110 13927 tablet_bootstrap.cc:654] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:09.143117 13928 ts_tablet_manager.cc:1397] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 22:40:09.143294 13928 raft_consensus.cc:348] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:09.143365 13928 raft_consensus.cc:374] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:09.143391 13928 raft_consensus.cc:729] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60952adeddc454d9fd5b7580bd840e3, State: Initialized, Role: FOLLOWER
I20260926 22:40:09.143432 13928 consensus_queue.cc:260] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:09.143522 13928 ts_tablet_manager.cc:1428] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:09.144058 13408 catalog_manager.cc:4841] TS afe9b2c1ad5f42b8a79ba0a7da24b8db (127.11.107.193:41121): tablet 6d128bec5f254357b1e865b679d235d3 (table test-table [id=fb85a895e7ac471f8899416774fd13dc]) successfully deleted
I20260926 22:40:09.144248 13545 tablet_service.cc:1467] Processing CreateTablet for tablet 971c61fb570b43cb9038572678b5c954 (DEFAULT_TABLE table=test-table [id=d9f4e19d8d4b4534a84a7a52ebda9f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:09.144344 13545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971c61fb570b43cb9038572678b5c954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:09.144577 13972 tablet_replica.cc:330] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd: stopping tablet replica
I20260926 22:40:09.144665 13972 raft_consensus.cc:2229] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:09.144711 13972 raft_consensus.cc:2258] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:09.145057 13927 tablet_bootstrap.cc:492] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd: No bootstrap required, opened a new log
I20260926 22:40:09.145097 13927 ts_tablet_manager.cc:1397] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:09.145237 13927 raft_consensus.cc:348] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:09.145295 13927 raft_consensus.cc:374] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:09.145313 13927 raft_consensus.cc:729] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30d6c22589c42c78e206a5206977cbd, State: Initialized, Role: FOLLOWER
I20260926 22:40:09.145349 13927 consensus_queue.cc:260] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:09.145442 13927 ts_tablet_manager.cc:1428] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:09.145750 13929 tablet_bootstrap.cc:492] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Bootstrap starting.
I20260926 22:40:09.145951 13973 tablet_replica.cc:330] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3: stopping tablet replica
I20260926 22:40:09.146044 13973 raft_consensus.cc:2229] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:09.146131 13973 raft_consensus.cc:2258] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:09.146157 13929 tablet_bootstrap.cc:654] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:09.146663 13973 ts_tablet_manager.cc:1905] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:09.146822 13972 ts_tablet_manager.cc:1905] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:09.148151 13972 ts_tablet_manager.cc:1918] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:09.148219 13972 log.cc:1196] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/6d128bec5f254357b1e865b679d235d3
I20260926 22:40:09.148242 13973 ts_tablet_manager.cc:1918] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:09.148312 13973 log.cc:1196] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/6d128bec5f254357b1e865b679d235d3
I20260926 22:40:09.148480 13972 ts_tablet_manager.cc:1939] T 6d128bec5f254357b1e865b679d235d3 P a30d6c22589c42c78e206a5206977cbd: Deleting consensus metadata
I20260926 22:40:09.148684 13973 ts_tablet_manager.cc:1939] T 6d128bec5f254357b1e865b679d235d3 P c60952adeddc454d9fd5b7580bd840e3: Deleting consensus metadata
I20260926 22:40:09.148890 13410 catalog_manager.cc:4841] TS a30d6c22589c42c78e206a5206977cbd (127.11.107.195:43679): tablet 6d128bec5f254357b1e865b679d235d3 (table test-table [id=fb85a895e7ac471f8899416774fd13dc]) successfully deleted
I20260926 22:40:09.149029 13409 catalog_manager.cc:4841] TS c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194:35437): tablet 6d128bec5f254357b1e865b679d235d3 (table test-table [id=fb85a895e7ac471f8899416774fd13dc]) successfully deleted
I20260926 22:40:09.149387 13929 tablet_bootstrap.cc:492] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db: No bootstrap required, opened a new log
I20260926 22:40:09.149452 13929 ts_tablet_manager.cc:1397] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 22:40:09.149597 13929 raft_consensus.cc:348] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:09.149686 13929 raft_consensus.cc:374] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:09.149714 13929 raft_consensus.cc:729] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe9b2c1ad5f42b8a79ba0a7da24b8db, State: Initialized, Role: FOLLOWER
I20260926 22:40:09.149757 13929 consensus_queue.cc:260] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:09.149878 13929 ts_tablet_manager.cc:1428] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:09.228230 13934 raft_consensus.cc:482] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:09.228346 13934 raft_consensus.cc:504] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:09.228708 13934 leader_election.cc:288] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194:35437), a30d6c22589c42c78e206a5206977cbd (127.11.107.195:43679)
I20260926 22:40:09.232744 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.229002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:54935 (local address 127.11.107.194:35437)
0926 22:40:09.229042 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:09.229044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:09.229203 (+ 159us) server_negotiation.cc:407] Connection header received
0926 22:40:09.229298 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:09.229299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:09.229308 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:09.229336 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:09.229582 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.230331 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.231834 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.232370 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.232416 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.232424 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:09.232628 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 22:40:09.232673 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:40:09.232769 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.229113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:60215 (local address 127.11.107.195:43679)
0926 22:40:09.229156 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:09.229159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:09.229279 (+ 120us) server_negotiation.cc:407] Connection header received
0926 22:40:09.229384 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:09.229385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:09.229397 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:09.229429 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:09.229652 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.230391 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.231840 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.232396 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.232443 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.232451 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:09.232667 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 22:40:09.232703 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:40:09.232993 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.228932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:43679 (local address 127.11.107.193:60215)
0926 22:40:09.229214 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:09.229224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:09.229304 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:09.229496 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:09.229497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:09.229499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:09.229572 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.229574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.230491 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.230492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.231737 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.231739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.232510 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.232511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.232585 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.232588 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:09.232878 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 22:40:09.232918 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":79,"threads_started":1}
I20260926 22:40:09.233233 13861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971c61fb570b43cb9038572678b5c954" candidate_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30d6c22589c42c78e206a5206977cbd" is_pre_election: true
I20260926 22:40:09.233326 13861 raft_consensus.cc:2454] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afe9b2c1ad5f42b8a79ba0a7da24b8db in term 0.
I20260926 22:40:09.233347 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.228858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35437 (local address 127.11.107.193:54935)
0926 22:40:09.229145 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:09.229155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:09.229229 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:09.229394 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:09.229395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:09.229396 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:09.229447 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.229448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.230430 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.230431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.231730 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.231733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.232482 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.232483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.232547 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.232552 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:40:09.233223 (+ 671us) client_negotiation.cc:240] Negotiation successful
0926 22:40:09.233265 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":47,"thread_start_us":58,"threads_started":1}
I20260926 22:40:09.233518 13514 leader_election.cc:302] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a30d6c22589c42c78e206a5206977cbd, afe9b2c1ad5f42b8a79ba0a7da24b8db; no voters:
I20260926 22:40:09.233574 13717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971c61fb570b43cb9038572678b5c954" candidate_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60952adeddc454d9fd5b7580bd840e3" is_pre_election: true
I20260926 22:40:09.233640 13934 raft_consensus.cc:2781] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:09.233665 13717 raft_consensus.cc:2454] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afe9b2c1ad5f42b8a79ba0a7da24b8db in term 0.
I20260926 22:40:09.233680 13934 raft_consensus.cc:482] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:09.233708 13934 raft_consensus.cc:3037] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:09.234628 13934 raft_consensus.cc:504] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:09.234762 13934 leader_election.cc:288] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [CANDIDATE]: Term 1 election: Requested vote from peers c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194:35437), a30d6c22589c42c78e206a5206977cbd (127.11.107.195:43679)
I20260926 22:40:09.234911 13717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971c61fb570b43cb9038572678b5c954" candidate_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60952adeddc454d9fd5b7580bd840e3"
I20260926 22:40:09.234929 13861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971c61fb570b43cb9038572678b5c954" candidate_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30d6c22589c42c78e206a5206977cbd"
I20260926 22:40:09.234979 13717 raft_consensus.cc:3037] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:09.234998 13861 raft_consensus.cc:3037] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:09.235677 13717 raft_consensus.cc:2454] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afe9b2c1ad5f42b8a79ba0a7da24b8db in term 1.
I20260926 22:40:09.235741 13861 raft_consensus.cc:2454] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afe9b2c1ad5f42b8a79ba0a7da24b8db in term 1.
I20260926 22:40:09.235841 13513 leader_election.cc:302] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afe9b2c1ad5f42b8a79ba0a7da24b8db, c60952adeddc454d9fd5b7580bd840e3; no voters:
I20260926 22:40:09.235989 13934 raft_consensus.cc:2781] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:09.236123 13934 raft_consensus.cc:686] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 LEADER]: Becoming Leader. State: Replica: afe9b2c1ad5f42b8a79ba0a7da24b8db, State: Running, Role: LEADER
I20260926 22:40:09.236176 13934 consensus_queue.cc:237] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:09.236835 13423 catalog_manager.cc:5495] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db reported cstate change: term changed from 0 to 1, leader changed from <none> to afe9b2c1ad5f42b8a79ba0a7da24b8db (127.11.107.193). New cstate: current_term: 1 leader_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } health_report { overall_health: UNKNOWN } } }
I20260926 22:40:09.265560 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.262982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.11.107.193:41121)
0926 22:40:09.263023 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:09.263025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:09.263034 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:09.263078 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:09.263079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:09.263104 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:09.263188 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:09.263270 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.264062 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.264761 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.265033 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.265107 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.265224 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:09.265348 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:09.265417 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:09.265439 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 22:40:09.265463 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:40:09.270658 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.268174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:32863 (local address 127.11.107.195:43679)
0926 22:40:09.268239 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:09.268242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:09.268294 (+ 52us) server_negotiation.cc:407] Connection header received
0926 22:40:09.268344 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:09.268345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:09.268355 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:09.268415 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:09.268548 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.269193 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.269921 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.270194 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.270269 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.270424 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:09.270456 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:09.270515 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:09.270570 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 22:40:09.270590 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 22:40:09.270860 13983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.268072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:43679 (local address 127.11.107.193:32863)
0926 22:40:09.268241 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:09.268251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:09.268304 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:09.268425 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:09.268426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:09.268427 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:40:09.268479 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.268480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.269271 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.269272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.269820 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.269821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.270286 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.270287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.270328 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.270340 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:40:09.270520 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:40:09.270521 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:40:09.270555 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 22:40:09.270588 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260926 22:40:09.272794 13717 raft_consensus.cc:1261] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Refusing update from remote peer afe9b2c1ad5f42b8a79ba0a7da24b8db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:09.272825 13861 raft_consensus.cc:1261] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Refusing update from remote peer afe9b2c1ad5f42b8a79ba0a7da24b8db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:09.273022 13934 consensus_queue.cc:1035] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:09.273149 13979 consensus_queue.cc:1035] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:09.280951 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.278417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:41121 (local address 127.11.107.195:53209)
0926 22:40:09.278464 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:09.278474 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:09.278508 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:09.278643 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:09.278644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:09.278645 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:40:09.278682 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.278684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.279489 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.279490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.280165 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:09.280167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.280649 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.280650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:09.280685 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.280688 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:40:09.280813 (+ 125us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:40:09.280814 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:40:09.280853 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 22:40:09.280902 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 22:40:09.280956 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:09.278457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:53209 (local address 127.11.107.193:41121)
0926 22:40:09.278490 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:09.278492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:09.278499 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:09.278561 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:09.278562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:09.278570 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:09.278627 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:09.278746 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.279421 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.280241 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:09.280523 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:09.280647 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:09.280736 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:09.280766 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:09.280808 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:09.280855 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 22:40:09.280891 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 22:40:09.281556 13955 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 22:40:10.801132 13423 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:10.801234 13423 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:10.802073 13423 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6: Sending DeleteTablet for 3 replicas of tablet 971c61fb570b43cb9038572678b5c954
I20260926 22:40:10.802340 13841 tablet_service.cc:1514] Processing DeleteTablet for tablet 971c61fb570b43cb9038572678b5c954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:10 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:51312
I20260926 22:40:10.802408 13545 tablet_service.cc:1514] Processing DeleteTablet for tablet 971c61fb570b43cb9038572678b5c954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:10 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:46916
I20260926 22:40:10.802767 14011 tablet_replica.cc:330] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db: stopping tablet replica
I20260926 22:40:10.802820 14011 raft_consensus.cc:2229] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:10.802906 14011 raft_consensus.cc:2258] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:10.803122 14011 ts_tablet_manager.cc:1905] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:10.803920 13701 tablet_service.cc:1514] Processing DeleteTablet for tablet 971c61fb570b43cb9038572678b5c954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:10 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:59180
I20260926 22:40:10.804163 14013 tablet_replica.cc:330] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3: stopping tablet replica
I20260926 22:40:10.804217 14013 raft_consensus.cc:2229] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:10.804247 14013 raft_consensus.cc:2258] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:10.804467 14013 ts_tablet_manager.cc:1905] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:10.804714 14011 ts_tablet_manager.cc:1918] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 22:40:10.804831 14011 log.cc:1196] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/wal/wals/971c61fb570b43cb9038572678b5c954
I20260926 22:40:10.805176 14011 ts_tablet_manager.cc:1939] T 971c61fb570b43cb9038572678b5c954 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Deleting consensus metadata
I20260926 22:40:10.805500 13408 catalog_manager.cc:4841] TS afe9b2c1ad5f42b8a79ba0a7da24b8db (127.11.107.193:41121): tablet 971c61fb570b43cb9038572678b5c954 (table test-table [id=d9f4e19d8d4b4534a84a7a52ebda9f63]) successfully deleted
I20260926 22:40:10.805727 14010 tablet_replica.cc:330] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd: stopping tablet replica
I20260926 22:40:10.805802 14010 raft_consensus.cc:2229] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:10.805882 14010 raft_consensus.cc:2258] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:10.806111 14010 ts_tablet_manager.cc:1905] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:10.806360 14013 ts_tablet_manager.cc:1918] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 22:40:10.806423 14013 log.cc:1196] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/971c61fb570b43cb9038572678b5c954
I20260926 22:40:10.806633 14013 ts_tablet_manager.cc:1939] T 971c61fb570b43cb9038572678b5c954 P c60952adeddc454d9fd5b7580bd840e3: Deleting consensus metadata
I20260926 22:40:10.806921 13409 catalog_manager.cc:4841] TS c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194:35437): tablet 971c61fb570b43cb9038572678b5c954 (table test-table [id=d9f4e19d8d4b4534a84a7a52ebda9f63]) successfully deleted
I20260926 22:40:10.807554 14010 ts_tablet_manager.cc:1918] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 22:40:10.807629 14010 log.cc:1196] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/971c61fb570b43cb9038572678b5c954
I20260926 22:40:10.807859 14010 ts_tablet_manager.cc:1939] T 971c61fb570b43cb9038572678b5c954 P a30d6c22589c42c78e206a5206977cbd: Deleting consensus metadata
I20260926 22:40:10.808236 13410 catalog_manager.cc:4841] TS a30d6c22589c42c78e206a5206977cbd (127.11.107.195:43679): tablet 971c61fb570b43cb9038572678b5c954 (table test-table [id=d9f4e19d8d4b4534a84a7a52ebda9f63]) successfully deleted
Sep 26 22:40:10 dist-test-slave-j8sq krb5kdc[13379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462410, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 22:40:10 dist-test-slave-j8sq krb5kdc[13379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462410, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:10.825587 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:10.818406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.11.107.254:38267)
0926 22:40:10.818547 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:10.818551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:10.818564 (+ 13us) server_negotiation.cc:407] Connection header received
0926 22:40:10.818619 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:10.818619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:10.818629 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:10.818738 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:10.819307 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:10.819876 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:10.820626 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:10.820834 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:10.820918 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:10.823255 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:10.823257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:10.823257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:10.823263 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:10.824549 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:10.824914 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:10.824915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:10.824915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:10.824964 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:10.825160 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:10.825160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:10.825161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:10.825255 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:10.825325 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:10.825463 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 22:40:10.825523 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":98,"threads_started":1}
I20260926 22:40:10.827025 13423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:10.827186 13423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:10.828557 13545 tablet_service.cc:1467] Processing CreateTablet for tablet d5022001c1984ff1addccc5f05a50853 (DEFAULT_TABLE table=test-table [id=b0e1876130fd4f62b347301a76fccdf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:10.828588 13701 tablet_service.cc:1467] Processing CreateTablet for tablet d5022001c1984ff1addccc5f05a50853 (DEFAULT_TABLE table=test-table [id=b0e1876130fd4f62b347301a76fccdf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:10.828670 13841 tablet_service.cc:1467] Processing CreateTablet for tablet d5022001c1984ff1addccc5f05a50853 (DEFAULT_TABLE table=test-table [id=b0e1876130fd4f62b347301a76fccdf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:10.828709 13545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5022001c1984ff1addccc5f05a50853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:10.828725 13701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5022001c1984ff1addccc5f05a50853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:10.828770 13841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5022001c1984ff1addccc5f05a50853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:10.829622 14021 tablet_bootstrap.cc:492] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3: Bootstrap starting.
I20260926 22:40:10.830047 14022 tablet_bootstrap.cc:492] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Bootstrap starting.
I20260926 22:40:10.830058 14021 tablet_bootstrap.cc:654] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:10.830323 14023 tablet_bootstrap.cc:492] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd: Bootstrap starting.
I20260926 22:40:10.830410 14022 tablet_bootstrap.cc:654] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:10.830714 14021 tablet_bootstrap.cc:492] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3: No bootstrap required, opened a new log
I20260926 22:40:10.830760 14023 tablet_bootstrap.cc:654] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:10.830828 14021 ts_tablet_manager.cc:1397] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:10.831001 14021 raft_consensus.cc:348] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.831067 14022 tablet_bootstrap.cc:492] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db: No bootstrap required, opened a new log
I20260926 22:40:10.831081 14021 raft_consensus.cc:374] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:10.831104 14021 raft_consensus.cc:729] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60952adeddc454d9fd5b7580bd840e3, State: Initialized, Role: FOLLOWER
I20260926 22:40:10.831110 14022 ts_tablet_manager.cc:1397] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:10.831189 14021 consensus_queue.cc:260] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.831234 14022 raft_consensus.cc:348] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.831343 14022 raft_consensus.cc:374] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:10.831377 14022 raft_consensus.cc:729] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe9b2c1ad5f42b8a79ba0a7da24b8db, State: Initialized, Role: FOLLOWER
I20260926 22:40:10.831372 14023 tablet_bootstrap.cc:492] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd: No bootstrap required, opened a new log
I20260926 22:40:10.831377 14021 ts_tablet_manager.cc:1428] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:10.831420 14023 ts_tablet_manager.cc:1397] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:10.831532 14023 raft_consensus.cc:348] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.831544 14022 consensus_queue.cc:260] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.831612 14023 raft_consensus.cc:374] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:10.831645 14023 raft_consensus.cc:729] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30d6c22589c42c78e206a5206977cbd, State: Initialized, Role: FOLLOWER
I20260926 22:40:10.831697 14023 consensus_queue.cc:260] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.831810 14023 ts_tablet_manager.cc:1428] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:10.831868 14022 ts_tablet_manager.cc:1428] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:10.845613 14003 raft_consensus.cc:482] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:10.845727 14003 raft_consensus.cc:504] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.845927 14003 leader_election.cc:288] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194:35437), a30d6c22589c42c78e206a5206977cbd (127.11.107.195:43679)
I20260926 22:40:10.846045 13717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5022001c1984ff1addccc5f05a50853" candidate_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60952adeddc454d9fd5b7580bd840e3" is_pre_election: true
I20260926 22:40:10.846112 13861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5022001c1984ff1addccc5f05a50853" candidate_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30d6c22589c42c78e206a5206977cbd" is_pre_election: true
I20260926 22:40:10.846185 13861 raft_consensus.cc:2454] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afe9b2c1ad5f42b8a79ba0a7da24b8db in term 0.
I20260926 22:40:10.846182 13717 raft_consensus.cc:2454] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afe9b2c1ad5f42b8a79ba0a7da24b8db in term 0.
I20260926 22:40:10.846339 13513 leader_election.cc:302] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afe9b2c1ad5f42b8a79ba0a7da24b8db, c60952adeddc454d9fd5b7580bd840e3; no voters:
I20260926 22:40:10.846472 14003 raft_consensus.cc:2781] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:10.846508 14003 raft_consensus.cc:482] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:10.846534 14003 raft_consensus.cc:3037] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:10.847266 14003 raft_consensus.cc:504] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.847402 14003 leader_election.cc:288] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [CANDIDATE]: Term 1 election: Requested vote from peers c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194:35437), a30d6c22589c42c78e206a5206977cbd (127.11.107.195:43679)
I20260926 22:40:10.847537 13861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5022001c1984ff1addccc5f05a50853" candidate_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30d6c22589c42c78e206a5206977cbd"
I20260926 22:40:10.847550 13717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5022001c1984ff1addccc5f05a50853" candidate_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60952adeddc454d9fd5b7580bd840e3"
I20260926 22:40:10.847616 13861 raft_consensus.cc:3037] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:10.847635 13717 raft_consensus.cc:3037] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:10.848191 13861 raft_consensus.cc:2454] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afe9b2c1ad5f42b8a79ba0a7da24b8db in term 1.
I20260926 22:40:10.848222 13717 raft_consensus.cc:2454] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afe9b2c1ad5f42b8a79ba0a7da24b8db in term 1.
I20260926 22:40:10.848378 13514 leader_election.cc:302] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a30d6c22589c42c78e206a5206977cbd, afe9b2c1ad5f42b8a79ba0a7da24b8db; no voters:
I20260926 22:40:10.848476 14003 raft_consensus.cc:2781] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:10.848543 14003 raft_consensus.cc:686] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 LEADER]: Becoming Leader. State: Replica: afe9b2c1ad5f42b8a79ba0a7da24b8db, State: Running, Role: LEADER
I20260926 22:40:10.848596 14003 consensus_queue.cc:237] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.849162 13423 catalog_manager.cc:5495] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db reported cstate change: term changed from 0 to 1, leader changed from <none> to afe9b2c1ad5f42b8a79ba0a7da24b8db (127.11.107.193). New cstate: current_term: 1 leader_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } health_report { overall_health: UNKNOWN } } }
I20260926 22:40:10.855810 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:10.853705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.11.107.193:41121)
0926 22:40:10.853863 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:10.853865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:10.853873 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:40:10.853894 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:10.853894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:10.853905 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:10.853970 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:10.854076 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:10.854545 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:10.855259 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:10.855443 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:10.855492 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:10.855627 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:10.855654 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:10.855690 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:10.855739 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 22:40:10.855755 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":98,"threads_started":1}
I20260926 22:40:10.857822 13717 raft_consensus.cc:1261] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Refusing update from remote peer afe9b2c1ad5f42b8a79ba0a7da24b8db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:10.857815 13861 raft_consensus.cc:1261] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Refusing update from remote peer afe9b2c1ad5f42b8a79ba0a7da24b8db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:10.858044 14003 consensus_queue.cc:1035] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:10.858140 14000 consensus_queue.cc:1035] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:10.867472 13423 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:10.867568 13423 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:10.868148 13423 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6: Sending DeleteTablet for 3 replicas of tablet d5022001c1984ff1addccc5f05a50853
I20260926 22:40:10.868367 13545 tablet_service.cc:1514] Processing DeleteTablet for tablet d5022001c1984ff1addccc5f05a50853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:10 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:46916
I20260926 22:40:10.868400 13841 tablet_service.cc:1514] Processing DeleteTablet for tablet d5022001c1984ff1addccc5f05a50853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:10 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:51312
I20260926 22:40:10.868484 14010 tablet_replica.cc:330] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd: stopping tablet replica
I20260926 22:40:10.868466 13701 tablet_service.cc:1514] Processing DeleteTablet for tablet d5022001c1984ff1addccc5f05a50853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:10 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:59180
I20260926 22:40:10.868535 14010 raft_consensus.cc:2229] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:10.868554 14013 tablet_replica.cc:330] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3: stopping tablet replica
I20260926 22:40:10.868561 14011 tablet_replica.cc:330] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db: stopping tablet replica
I20260926 22:40:10.868571 14010 raft_consensus.cc:2258] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:10.868608 14011 raft_consensus.cc:2229] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:10.868613 14013 raft_consensus.cc:2229] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:10.868649 14013 raft_consensus.cc:2258] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:10.868652 14011 raft_consensus.cc:2258] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:10.868582 13423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:10.868710 13423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:10.868947 14011 ts_tablet_manager.cc:1905] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:10.869032 14010 ts_tablet_manager.cc:1905] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:10.869030 14013 ts_tablet_manager.cc:1905] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:10.869771 13701 tablet_service.cc:1467] Processing CreateTablet for tablet 9798f6a1b18c4c709b8edd180f81f398 (DEFAULT_TABLE table=test-table [id=53d9011802674cbabeb5b4205e5916c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:10.869905 13701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9798f6a1b18c4c709b8edd180f81f398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:10.870771 13841 tablet_service.cc:1467] Processing CreateTablet for tablet 9798f6a1b18c4c709b8edd180f81f398 (DEFAULT_TABLE table=test-table [id=53d9011802674cbabeb5b4205e5916c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:10.870805 14010 ts_tablet_manager.cc:1918] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:10.870791 14011 ts_tablet_manager.cc:1918] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:10.870848 14010 log.cc:1196] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/d5022001c1984ff1addccc5f05a50853
I20260926 22:40:10.870865 14011 log.cc:1196] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-0/wal/wals/d5022001c1984ff1addccc5f05a50853
I20260926 22:40:10.870877 13841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9798f6a1b18c4c709b8edd180f81f398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:10.871094 14011 ts_tablet_manager.cc:1939] T d5022001c1984ff1addccc5f05a50853 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Deleting consensus metadata
I20260926 22:40:10.871104 14010 ts_tablet_manager.cc:1939] T d5022001c1984ff1addccc5f05a50853 P a30d6c22589c42c78e206a5206977cbd: Deleting consensus metadata
I20260926 22:40:10.871377 13545 tablet_service.cc:1467] Processing CreateTablet for tablet 9798f6a1b18c4c709b8edd180f81f398 (DEFAULT_TABLE table=test-table [id=53d9011802674cbabeb5b4205e5916c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:10.871403 13410 catalog_manager.cc:4841] TS a30d6c22589c42c78e206a5206977cbd (127.11.107.195:43679): tablet d5022001c1984ff1addccc5f05a50853 (table test-table [id=b0e1876130fd4f62b347301a76fccdf7]) successfully deleted
I20260926 22:40:10.871418 13408 catalog_manager.cc:4841] TS afe9b2c1ad5f42b8a79ba0a7da24b8db (127.11.107.193:41121): tablet d5022001c1984ff1addccc5f05a50853 (table test-table [id=b0e1876130fd4f62b347301a76fccdf7]) successfully deleted
I20260926 22:40:10.871474 13545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9798f6a1b18c4c709b8edd180f81f398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:10.871544 14013 ts_tablet_manager.cc:1918] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:10.871593 14013 log.cc:1196] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/d5022001c1984ff1addccc5f05a50853
I20260926 22:40:10.871798 14013 ts_tablet_manager.cc:1939] T d5022001c1984ff1addccc5f05a50853 P c60952adeddc454d9fd5b7580bd840e3: Deleting consensus metadata
I20260926 22:40:10.872076 13409 catalog_manager.cc:4841] TS c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194:35437): tablet d5022001c1984ff1addccc5f05a50853 (table test-table [id=b0e1876130fd4f62b347301a76fccdf7]) successfully deleted
I20260926 22:40:10.872506 14022 tablet_bootstrap.cc:492] T 9798f6a1b18c4c709b8edd180f81f398 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Bootstrap starting.
I20260926 22:40:10.872792 14021 tablet_bootstrap.cc:492] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3: Bootstrap starting.
I20260926 22:40:10.872820 14023 tablet_bootstrap.cc:492] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd: Bootstrap starting.
I20260926 22:40:10.872840 14022 tablet_bootstrap.cc:654] T 9798f6a1b18c4c709b8edd180f81f398 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:10.873087 14021 tablet_bootstrap.cc:654] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:10.873119 14023 tablet_bootstrap.cc:654] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:10.873441 14022 tablet_bootstrap.cc:492] T 9798f6a1b18c4c709b8edd180f81f398 P afe9b2c1ad5f42b8a79ba0a7da24b8db: No bootstrap required, opened a new log
I20260926 22:40:10.873495 14022 ts_tablet_manager.cc:1397] T 9798f6a1b18c4c709b8edd180f81f398 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:10.873669 14022 raft_consensus.cc:348] T 9798f6a1b18c4c709b8edd180f81f398 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.873769 14022 raft_consensus.cc:374] T 9798f6a1b18c4c709b8edd180f81f398 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:10.873801 14022 raft_consensus.cc:729] T 9798f6a1b18c4c709b8edd180f81f398 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe9b2c1ad5f42b8a79ba0a7da24b8db, State: Initialized, Role: FOLLOWER
I20260926 22:40:10.873884 14022 consensus_queue.cc:260] T 9798f6a1b18c4c709b8edd180f81f398 P afe9b2c1ad5f42b8a79ba0a7da24b8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.873979 14022 ts_tablet_manager.cc:1428] T 9798f6a1b18c4c709b8edd180f81f398 P afe9b2c1ad5f42b8a79ba0a7da24b8db: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:10.874230 14023 tablet_bootstrap.cc:492] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd: No bootstrap required, opened a new log
I20260926 22:40:10.874284 14023 ts_tablet_manager.cc:1397] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:10.874413 14021 tablet_bootstrap.cc:492] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3: No bootstrap required, opened a new log
I20260926 22:40:10.874452 14021 ts_tablet_manager.cc:1397] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:10.874419 14023 raft_consensus.cc:348] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.874482 14023 raft_consensus.cc:374] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:10.874505 14023 raft_consensus.cc:729] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30d6c22589c42c78e206a5206977cbd, State: Initialized, Role: FOLLOWER
I20260926 22:40:10.874540 14023 consensus_queue.cc:260] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.874547 14021 raft_consensus.cc:348] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.874624 14023 ts_tablet_manager.cc:1428] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:10.874614 14021 raft_consensus.cc:374] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:10.874651 14021 raft_consensus.cc:729] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60952adeddc454d9fd5b7580bd840e3, State: Initialized, Role: FOLLOWER
I20260926 22:40:10.874698 14021 consensus_queue.cc:260] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:10.874809 14021 ts_tablet_manager.cc:1428] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:11.000313 14005 raft_consensus.cc:482] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:11.000422 14005 raft_consensus.cc:504] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:11.000604 14005 leader_election.cc:288] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afe9b2c1ad5f42b8a79ba0a7da24b8db (127.11.107.193:41121), c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194:35437)
I20260926 22:40:11.000803 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9798f6a1b18c4c709b8edd180f81f398" candidate_uuid: "a30d6c22589c42c78e206a5206977cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" is_pre_election: true
I20260926 22:40:11.000824 13717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9798f6a1b18c4c709b8edd180f81f398" candidate_uuid: "a30d6c22589c42c78e206a5206977cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60952adeddc454d9fd5b7580bd840e3" is_pre_election: true
I20260926 22:40:11.000929 13581 raft_consensus.cc:2454] T 9798f6a1b18c4c709b8edd180f81f398 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a30d6c22589c42c78e206a5206977cbd in term 0.
I20260926 22:40:11.000924 13717 raft_consensus.cc:2454] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a30d6c22589c42c78e206a5206977cbd in term 0.
I20260926 22:40:11.001089 13792 leader_election.cc:302] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a30d6c22589c42c78e206a5206977cbd, afe9b2c1ad5f42b8a79ba0a7da24b8db; no voters:
I20260926 22:40:11.001230 14005 raft_consensus.cc:2781] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:11.001276 14005 raft_consensus.cc:482] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:11.001299 14005 raft_consensus.cc:3037] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:11.002059 14005 raft_consensus.cc:504] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:11.002199 14005 leader_election.cc:288] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [CANDIDATE]: Term 1 election: Requested vote from peers afe9b2c1ad5f42b8a79ba0a7da24b8db (127.11.107.193:41121), c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194:35437)
I20260926 22:40:11.002343 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9798f6a1b18c4c709b8edd180f81f398" candidate_uuid: "a30d6c22589c42c78e206a5206977cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db"
I20260926 22:40:11.002391 13717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9798f6a1b18c4c709b8edd180f81f398" candidate_uuid: "a30d6c22589c42c78e206a5206977cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60952adeddc454d9fd5b7580bd840e3"
I20260926 22:40:11.002442 13581 raft_consensus.cc:3037] T 9798f6a1b18c4c709b8edd180f81f398 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:11.002475 13717 raft_consensus.cc:3037] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:11.003044 13717 raft_consensus.cc:2454] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a30d6c22589c42c78e206a5206977cbd in term 1.
I20260926 22:40:11.003134 13581 raft_consensus.cc:2454] T 9798f6a1b18c4c709b8edd180f81f398 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a30d6c22589c42c78e206a5206977cbd in term 1.
I20260926 22:40:11.003224 13793 leader_election.cc:302] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a30d6c22589c42c78e206a5206977cbd, c60952adeddc454d9fd5b7580bd840e3; no voters:
I20260926 22:40:11.003340 14005 raft_consensus.cc:2781] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:11.003391 14005 raft_consensus.cc:686] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [term 1 LEADER]: Becoming Leader. State: Replica: a30d6c22589c42c78e206a5206977cbd, State: Running, Role: LEADER
I20260926 22:40:11.003439 14005 consensus_queue.cc:237] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } }
I20260926 22:40:11.003979 13423 catalog_manager.cc:5495] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd reported cstate change: term changed from 0 to 1, leader changed from <none> to a30d6c22589c42c78e206a5206977cbd (127.11.107.195). New cstate: current_term: 1 leader_uuid: "a30d6c22589c42c78e206a5206977cbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a30d6c22589c42c78e206a5206977cbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 43679 } health_report { overall_health: HEALTHY } } }
I20260926 22:40:11.023739 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:11.021560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.11.107.195:43679)
0926 22:40:11.021728 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:11.021730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:11.021736 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:40:11.021754 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:11.021755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:11.021764 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:11.021819 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:11.021985 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:11.022457 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:11.023115 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:11.023302 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:11.023350 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:11.023468 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:11.023494 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:11.023532 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:11.023582 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 22:40:11.023597 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":100,"threads_started":1}
I20260926 22:40:11.028005 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:11.025727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:58073 (local address 127.11.107.195:43679)
0926 22:40:11.025759 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:11.025761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:11.025768 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:11.025830 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:11.025831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:11.025860 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:11.025911 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:11.026008 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:11.026425 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:11.027291 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:11.027490 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:11.027564 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:11.027691 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:11.027724 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:11.027785 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:11.027844 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 22:40:11.027935 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":18}
I20260926 22:40:11.028012 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:11.025585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:43679 (local address 127.11.107.195:58073)
0926 22:40:11.025717 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:11.025727 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:11.025776 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:11.025915 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:11.025916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:11.025917 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:40:11.025953 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:11.025954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:11.026509 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:11.026510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:11.027196 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:11.027199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:11.027560 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:11.027561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:11.027612 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:11.027620 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:40:11.027785 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:40:11.027787 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:40:11.027840 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 22:40:11.027922 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260926 22:40:11.030099 13581 raft_consensus.cc:1261] T 9798f6a1b18c4c709b8edd180f81f398 P afe9b2c1ad5f42b8a79ba0a7da24b8db [term 1 FOLLOWER]: Refusing update from remote peer a30d6c22589c42c78e206a5206977cbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:11.030190 13721 raft_consensus.cc:1261] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Refusing update from remote peer a30d6c22589c42c78e206a5206977cbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:11.030405 13949 consensus_queue.cc:1035] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60952adeddc454d9fd5b7580bd840e3" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:11.030545 13949 consensus_queue.cc:1035] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "afe9b2c1ad5f42b8a79ba0a7da24b8db" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 22:40:11.035321 13988 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 22:40:12.556178 13424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:12.556294 13424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:12.557088 13424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6307e1b4f054e04a85b9426decc21a6: Sending DeleteTablet for 3 replicas of tablet 9798f6a1b18c4c709b8edd180f81f398
I20260926 22:40:12.557340 13545 tablet_service.cc:1514] Processing DeleteTablet for tablet 9798f6a1b18c4c709b8edd180f81f398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:12 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:46916
I20260926 22:40:12.557391 13841 tablet_service.cc:1514] Processing DeleteTablet for tablet 9798f6a1b18c4c709b8edd180f81f398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:12 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:51312
I20260926 22:40:12.557415 13701 tablet_service.cc:1514] Processing DeleteTablet for tablet 9798f6a1b18c4c709b8edd180f81f398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:12 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:59180
I20260926 22:40:12.557574 14066 tablet_replica.cc:330] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3: stopping tablet replica
I20260926 22:40:12.557570 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 13496
I20260926 22:40:12.557644 14066 raft_consensus.cc:2229] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:12.558346 14066 raft_consensus.cc:2258] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:12.558564 14066 ts_tablet_manager.cc:1905] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:12.559999 14066 ts_tablet_manager.cc:1918] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 22:40:12.560048 14066 log.cc:1196] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/9798f6a1b18c4c709b8edd180f81f398
I20260926 22:40:12.560269 14066 ts_tablet_manager.cc:1939] T 9798f6a1b18c4c709b8edd180f81f398 P c60952adeddc454d9fd5b7580bd840e3: Deleting consensus metadata
I20260926 22:40:12.560577 13409 catalog_manager.cc:4841] TS c60952adeddc454d9fd5b7580bd840e3 (127.11.107.194:35437): tablet 9798f6a1b18c4c709b8edd180f81f398 (table test-table [id=53d9011802674cbabeb5b4205e5916c3]) successfully deleted
I20260926 22:40:12.565057 14065 tablet_replica.cc:330] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd: stopping tablet replica
I20260926 22:40:12.565148 14065 raft_consensus.cc:2229] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:12.565240 14065 raft_consensus.cc:2258] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 22:40:12.565435 13408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:41121 (error 108)
I20260926 22:40:12.565742 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 13636
I20260926 22:40:12.566401 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:12.565787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:41121 (local address 127.0.0.1:46922)
0926 22:40:12.566312 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:12.566361 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:41121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":107,"threads_started":1}
W20260926 22:40:12.566506 13408 catalog_manager.cc:4584] TS afe9b2c1ad5f42b8a79ba0a7da24b8db (127.11.107.193:41121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9798f6a1b18c4c709b8edd180f81f398: Network error: Client connection negotiation failed: client connection to 127.11.107.193:41121: connect: Connection refused (error 111)
I20260926 22:40:12.567543 14065 ts_tablet_manager.cc:1905] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:12.569492 14065 ts_tablet_manager.cc:1918] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 22:40:12.569558 14065 log.cc:1196] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/9798f6a1b18c4c709b8edd180f81f398
I20260926 22:40:12.569806 14065 ts_tablet_manager.cc:1939] T 9798f6a1b18c4c709b8edd180f81f398 P a30d6c22589c42c78e206a5206977cbd: Deleting consensus metadata
I20260926 22:40:12.570204 13410 catalog_manager.cc:4841] TS a30d6c22589c42c78e206a5206977cbd (127.11.107.195:43679): tablet 9798f6a1b18c4c709b8edd180f81f398 (table test-table [id=53d9011802674cbabeb5b4205e5916c3]) successfully deleted
I20260926 22:40:12.572376 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 13776
I20260926 22:40:12.578766 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 13392
2026-09-26T22:40:12Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5510 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:12 dist-test-slave-j8sq krb5kdc[14071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:12 dist-test-slave-j8sq krb5kdc[14071](info): set up 2 sockets
Sep 26 22:40:12 dist-test-slave-j8sq krb5kdc[14071](info): commencing operation
krb5kdc: starting...
W20260926 22:40:13.634641 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:13 dist-test-slave-j8sq krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:40:13.748795 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:41449
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:41449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:13.827229 14084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:13.827448 14084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:13.827485 14084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:13.828612 14084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:13.828680 14084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:13.828706 14084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:13.828737 14084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:13.828760 14084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:13.830272 14084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:41449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:41449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:13.830536 14084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:13.830785 14084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:13.835554 14090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:13.835683 14084 server_base.cc:1034] running on GCE node
W20260926 22:40:13.835582 14089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:13.835755 14092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:13.836191 14084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:13.836582 14084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:13.837764 14084 hybrid_clock.cc:648] HybridClock initialized: now 1790462413837751 us; error 28 us; skew 500 ppm
Sep 26 22:40:13 dist-test-slave-j8sq krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462413, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:13.840840 14084 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:40:13.841243 14084 webserver.cc:460] Webserver started at http://127.11.107.254:46617/ using document root <none> and password file <none>
I20260926 22:40:13.841468 14084 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:13.841533 14084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:13.841681 14084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:13.842814 14084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "5df1a371e4724c3caf3de6e55b3c34ef"
format_stamp: "Formatted at 2026-09-26 22:40:13 on dist-test-slave-j8sq"
server_key: "46a5fb587102f63aebbcaf120be65290"
server_key_iv: "0e487f5ec7eef4bf5bcc63e340cda87f"
server_key_version: "encryptionkey@0"
I20260926 22:40:13.843180 14084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "5df1a371e4724c3caf3de6e55b3c34ef"
format_stamp: "Formatted at 2026-09-26 22:40:13 on dist-test-slave-j8sq"
server_key: "46a5fb587102f63aebbcaf120be65290"
server_key_iv: "0e487f5ec7eef4bf5bcc63e340cda87f"
server_key_version: "encryptionkey@0"
I20260926 22:40:13.844476 14084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:13.845204 14099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:13.845376 14084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:13.845471 14084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "5df1a371e4724c3caf3de6e55b3c34ef"
format_stamp: "Formatted at 2026-09-26 22:40:13 on dist-test-slave-j8sq"
server_key: "46a5fb587102f63aebbcaf120be65290"
server_key_iv: "0e487f5ec7eef4bf5bcc63e340cda87f"
server_key_version: "encryptionkey@0"
I20260926 22:40:13.845551 14084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:13.856422 14084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:13.858810 14084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:13.858971 14084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:13.862958 14084 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:41449
I20260926 22:40:13.862989 14151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:41449 every 8 connection(s)
I20260926 22:40:13.863374 14084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:13.864276 14152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:13.866442 14152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef: Bootstrap starting.
I20260926 22:40:13.866958 14152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:13.867194 14152 log.cc:824] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:13.867833 14152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef: No bootstrap required, opened a new log
I20260926 22:40:13.869335 14152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df1a371e4724c3caf3de6e55b3c34ef" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41449 } }
I20260926 22:40:13.869467 14152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:13.869489 14152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df1a371e4724c3caf3de6e55b3c34ef, State: Initialized, Role: FOLLOWER
I20260926 22:40:13.869622 14152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df1a371e4724c3caf3de6e55b3c34ef" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41449 } }
I20260926 22:40:13.869704 14152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:13.869732 14152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:13.869767 14152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:13.870302 14152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df1a371e4724c3caf3de6e55b3c34ef" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41449 } }
I20260926 22:40:13.870386 14152 leader_election.cc:302] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5df1a371e4724c3caf3de6e55b3c34ef; no voters:
I20260926 22:40:13.870505 14152 leader_election.cc:288] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:13.870570 14155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:13.870751 14152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:13.870766 14155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [term 1 LEADER]: Becoming Leader. State: Replica: 5df1a371e4724c3caf3de6e55b3c34ef, State: Running, Role: LEADER
I20260926 22:40:13.870944 14155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df1a371e4724c3caf3de6e55b3c34ef" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41449 } }
I20260926 22:40:13.871492 14157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5df1a371e4724c3caf3de6e55b3c34ef. Latest consensus state: current_term: 1 leader_uuid: "5df1a371e4724c3caf3de6e55b3c34ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df1a371e4724c3caf3de6e55b3c34ef" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41449 } } }
I20260926 22:40:13.871567 14157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:13.871762 14156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5df1a371e4724c3caf3de6e55b3c34ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df1a371e4724c3caf3de6e55b3c34ef" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41449 } } }
I20260926 22:40:13.871860 14156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:13.872042 14164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:13.872620 14164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:13.872816 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 14084
I20260926 22:40:13.872908 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:13.873075 11695 external_mini_cluster.cc:1428] Setting key 6c8fd1725b28dc10c196853821cc78ba
I20260926 22:40:13.874157 14164 catalog_manager.cc:1347] Generated new cluster ID: fbb54c13bb3c4fd9b015d9b12eb17fb2
I20260926 22:40:13.874205 14164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:40:13 dist-test-slave-j8sq krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:13.882025 14164 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:13.882673 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:13.873677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57736 (local address 127.11.107.254:41449)
0926 22:40:13.873929 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:13.873937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:13.873960 (+ 23us) server_negotiation.cc:407] Connection header received
0926 22:40:13.874049 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:13.874075 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:13.874135 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:13.874291 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:13.874905 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:13.875806 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:13.876577 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:13.877013 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:13.877142 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:13.879463 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:13.879467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:13.879467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:13.879478 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:13.881142 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:13.881520 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:13.881523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:13.881523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:13.881609 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:13.881820 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:13.881821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:13.881821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:13.882120 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:13.882296 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:13.882363 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 22:40:13.882495 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":73,"threads_started":1}
I20260926 22:40:13.882997 14164 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:13.886065 14164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5df1a371e4724c3caf3de6e55b3c34ef: Generated new TSK 0
I20260926 22:40:13.886228 14164 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:40:13.917156 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:41449
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:14.001832 14178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:14.002177 14178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:14.002228 14178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:14.003387 14178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:14.003451 14178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:14.003499 14178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:14.004899 14178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:41449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:14.005149 14178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:14.005374 14178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:14.010967 14183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:14.011063 14186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:14.010958 14184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:14.011382 14178 server_base.cc:1034] running on GCE node
I20260926 22:40:14.011605 14178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:14.011838 14178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:14.013006 14178 hybrid_clock.cc:648] HybridClock initialized: now 1790462414012991 us; error 28 us; skew 500 ppm
Sep 26 22:40:14 dist-test-slave-j8sq krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462414, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:14.015498 14178 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:40:14.015955 14178 webserver.cc:460] Webserver started at http://127.11.107.193:38075/ using document root <none> and password file <none>
I20260926 22:40:14.016189 14178 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:14.016237 14178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:14.016364 14178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:14.017417 14178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "9ee593249ec84237b67763bbf7826acc"
format_stamp: "Formatted at 2026-09-26 22:40:14 on dist-test-slave-j8sq"
server_key: "ade27ba99da4124b05c468872b1b3916"
server_key_iv: "87c8d7ab0364318e633e7809889d8866"
server_key_version: "encryptionkey@0"
I20260926 22:40:14.017769 14178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "9ee593249ec84237b67763bbf7826acc"
format_stamp: "Formatted at 2026-09-26 22:40:14 on dist-test-slave-j8sq"
server_key: "ade27ba99da4124b05c468872b1b3916"
server_key_iv: "87c8d7ab0364318e633e7809889d8866"
server_key_version: "encryptionkey@0"
I20260926 22:40:14.019106 14178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:14.019861 14193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:14.020066 14178 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:14.020180 14178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "9ee593249ec84237b67763bbf7826acc"
format_stamp: "Formatted at 2026-09-26 22:40:14 on dist-test-slave-j8sq"
server_key: "ade27ba99da4124b05c468872b1b3916"
server_key_iv: "87c8d7ab0364318e633e7809889d8866"
server_key_version: "encryptionkey@0"
I20260926 22:40:14.020326 14178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:14.034884 14178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:14.037241 14178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:14.037416 14178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:14.037652 14178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:14.038035 14178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:14.038090 14178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:14.038128 14178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:14.038154 14178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:14.043785 14178 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:38769
I20260926 22:40:14.043860 14306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:38769 every 8 connection(s)
I20260926 22:40:14.044191 14178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
Sep 26 22:40:14 dist-test-slave-j8sq krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462414, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:14.052016 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 14178
I20260926 22:40:14.052124 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:14.052342 11695 external_mini_cluster.cc:1428] Setting key 87c85183b78e38612fee42ad0131133c
I20260926 22:40:14.054265 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:14.044951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:47423 (local address 127.11.107.254:41449)
0926 22:40:14.044997 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:14.045000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:14.045224 (+ 224us) server_negotiation.cc:407] Connection header received
0926 22:40:14.045677 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:14.045678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:14.045691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:14.045740 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:14.046695 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:14.047179 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:14.048149 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:14.048332 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:14.048384 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:14.050646 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:14.050648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:14.050648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:14.050656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:14.052392 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:14.052895 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:14.052896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:14.052896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:14.052960 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:14.053360 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:14.053361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:14.053362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:14.053533 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:14.053617 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:14.054064 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 22:40:14.054170 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:40:14.054944 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:14.044940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:41449 (local address 127.11.107.193:47423)
0926 22:40:14.045097 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:14.045132 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:14.045590 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:14.045797 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:14.045800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:14.046424 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:14.046611 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:14.046616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:14.047255 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:14.047255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:14.048074 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:14.048077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:14.048397 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:14.048398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:14.048496 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:14.048920 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:14.048928 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:14.050558 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:14.052466 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:14.052478 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:14.052480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:14.052825 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:14.053104 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:14.053105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:14.053105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:14.053292 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:14.053618 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:14.053625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:14.053823 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 22:40:14.054497 (+ 674us) client_negotiation.cc:240] Negotiation successful
0926 22:40:14.054688 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260926 22:40:14.055783 14307 heartbeater.cc:347] Connected to a master server at 127.11.107.254:41449
I20260926 22:40:14.055881 14307 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:14.056048 14307 heartbeater.cc:511] Master 127.11.107.254:41449 requested a full tablet report, sending...
I20260926 22:40:14.056587 14116 ts_manager.cc:194] Registered new tserver with Master: 9ee593249ec84237b67763bbf7826acc (127.11.107.193:38769)
I20260926 22:40:14.057536 14116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:47423
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:40:14.076797 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:41449
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:14.155848 14312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:14.156075 14312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:14.156097 14312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:14.157212 14312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:14.157246 14312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:14.157271 14312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:14.158722 14312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:41449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:14.158897 14312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:14.159058 14312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:14.163688 14318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:14.163803 14312 server_base.cc:1034] running on GCE node
W20260926 22:40:14.163759 14317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:14.163688 14320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:14.164183 14312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:14.164431 14312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:14.165588 14312 hybrid_clock.cc:648] HybridClock initialized: now 1790462414165583 us; error 36 us; skew 500 ppm
Sep 26 22:40:14 dist-test-slave-j8sq krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462414, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:14.168040 14312 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:40:14.168453 14312 webserver.cc:460] Webserver started at http://127.11.107.194:46141/ using document root <none> and password file <none>
I20260926 22:40:14.168669 14312 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:14.168733 14312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:14.168879 14312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:14.169941 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "5e39299c085a40d2b88451e86e0455b0"
format_stamp: "Formatted at 2026-09-26 22:40:14 on dist-test-slave-j8sq"
server_key: "7ff3c28a9242dcbcb2859109e5e64aec"
server_key_iv: "79854d4cd81dee78cce035ca50324e6f"
server_key_version: "encryptionkey@0"
I20260926 22:40:14.170288 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "5e39299c085a40d2b88451e86e0455b0"
format_stamp: "Formatted at 2026-09-26 22:40:14 on dist-test-slave-j8sq"
server_key: "7ff3c28a9242dcbcb2859109e5e64aec"
server_key_iv: "79854d4cd81dee78cce035ca50324e6f"
server_key_version: "encryptionkey@0"
I20260926 22:40:14.171731 14312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:40:14.172492 14327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:14.172665 14312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:14.172758 14312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "5e39299c085a40d2b88451e86e0455b0"
format_stamp: "Formatted at 2026-09-26 22:40:14 on dist-test-slave-j8sq"
server_key: "7ff3c28a9242dcbcb2859109e5e64aec"
server_key_iv: "79854d4cd81dee78cce035ca50324e6f"
server_key_version: "encryptionkey@0"
I20260926 22:40:14.172839 14312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:14.187968 14312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:14.190281 14312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:14.190440 14312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:14.190670 14312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:14.190987 14312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:14.191042 14312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:14.191092 14312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:14.191121 14312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:14.196557 14312 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:33001
I20260926 22:40:14.196604 14440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:33001 every 8 connection(s)
I20260926 22:40:14.196920 14312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:40:14.201073 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 14312
I20260926 22:40:14.201179 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:14.201385 11695 external_mini_cluster.cc:1428] Setting key 55d9e8a0b868f69698afbb23cfcc60c6
Sep 26 22:40:14 dist-test-slave-j8sq krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462414, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:14.206262 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:14.197589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:53585 (local address 127.11.107.254:41449)
0926 22:40:14.197637 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:14.197640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:14.197874 (+ 234us) server_negotiation.cc:407] Connection header received
0926 22:40:14.198369 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:14.198370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:14.198382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:14.198439 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:14.199301 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:14.199799 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:14.200768 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:14.200943 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:14.200993 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:14.203299 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:14.203301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:14.203302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:14.203310 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:14.204848 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:14.205229 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:14.205230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:14.205231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:14.205292 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:14.205539 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:14.205540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:14.205541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:14.205684 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:14.205756 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:14.206033 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 22:40:14.206131 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:40:14.206272 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:14.197576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:41449 (local address 127.11.107.194:53585)
0926 22:40:14.197743 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:14.197775 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:14.198266 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:14.198505 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:14.198508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:14.199049 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:14.199228 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:14.199233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:14.199869 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:14.199870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:14.200701 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:14.200703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:14.201030 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:14.201032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:14.201188 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:14.201714 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:14.201722 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:14.203205 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:14.204919 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:14.204924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:14.204925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:14.205145 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:14.205370 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:14.205370 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:14.205371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:14.205476 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:14.205749 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:14.205753 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:14.205906 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 22:40:14.206000 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 22:40:14.206112 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260926 22:40:14.207085 14441 heartbeater.cc:347] Connected to a master server at 127.11.107.254:41449
I20260926 22:40:14.207201 14441 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:14.207367 14441 heartbeater.cc:511] Master 127.11.107.254:41449 requested a full tablet report, sending...
I20260926 22:40:14.207729 14116 ts_manager.cc:194] Registered new tserver with Master: 5e39299c085a40d2b88451e86e0455b0 (127.11.107.194:33001)
I20260926 22:40:14.208087 14116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:53585
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:40:14.225785 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:41449
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:14.303836 14446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:14.304023 14446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:14.304045 14446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:14.305102 14446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:14.305133 14446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:14.305158 14446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:14.306499 14446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:41449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:14.306674 14446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:14.306846 14446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:14.311589 14451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:14.311601 14452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:14.311601 14454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:14.311735 14446 server_base.cc:1034] running on GCE node
I20260926 22:40:14.311998 14446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:14.312201 14446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:14.313383 14446 hybrid_clock.cc:648] HybridClock initialized: now 1790462414313374 us; error 27 us; skew 500 ppm
Sep 26 22:40:14 dist-test-slave-j8sq krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462414, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:14.315706 14446 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:40:14.316131 14446 webserver.cc:460] Webserver started at http://127.11.107.195:35527/ using document root <none> and password file <none>
I20260926 22:40:14.316359 14446 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:14.316423 14446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:14.316567 14446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:14.317528 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "35a552a17a5e4191a7e8cd4d4206b239"
format_stamp: "Formatted at 2026-09-26 22:40:14 on dist-test-slave-j8sq"
server_key: "73eb444d55daf706214e9e35b38a4aad"
server_key_iv: "0511236f5f62e49d8dc945369dfb47cd"
server_key_version: "encryptionkey@0"
I20260926 22:40:14.317936 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "35a552a17a5e4191a7e8cd4d4206b239"
format_stamp: "Formatted at 2026-09-26 22:40:14 on dist-test-slave-j8sq"
server_key: "73eb444d55daf706214e9e35b38a4aad"
server_key_iv: "0511236f5f62e49d8dc945369dfb47cd"
server_key_version: "encryptionkey@0"
I20260926 22:40:14.319191 14446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:14.319883 14461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:14.320051 14446 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:14.320158 14446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "35a552a17a5e4191a7e8cd4d4206b239"
format_stamp: "Formatted at 2026-09-26 22:40:14 on dist-test-slave-j8sq"
server_key: "73eb444d55daf706214e9e35b38a4aad"
server_key_iv: "0511236f5f62e49d8dc945369dfb47cd"
server_key_version: "encryptionkey@0"
I20260926 22:40:14.320236 14446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:14.331982 14446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:14.333703 14446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:14.333933 14446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:14.334154 14446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:14.334448 14446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:14.334501 14446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:14.334534 14446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:14.334561 14446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:14.340111 14446 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:42561
I20260926 22:40:14.340158 14574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:42561 every 8 connection(s)
I20260926 22:40:14.340474 14446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:40:14 dist-test-slave-j8sq krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462414, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:14.349025 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:14.341166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:50577 (local address 127.11.107.254:41449)
0926 22:40:14.341241 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:14.341243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:14.341401 (+ 158us) server_negotiation.cc:407] Connection header received
0926 22:40:14.341836 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:14.341857 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:14.341868 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:14.341912 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:14.342776 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:14.343254 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:14.344178 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:14.344352 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:14.344401 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:14.346601 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:14.346602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:14.346602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:14.346608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:14.347824 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:14.348176 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:14.348177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:14.348177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:14.348224 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:14.348470 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:14.348471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:14.348471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:14.348567 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:14.348616 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:14.348903 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 22:40:14.348962 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 22:40:14.349172 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:14.341114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:41449 (local address 127.11.107.195:50577)
0926 22:40:14.341290 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:14.341322 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:14.341757 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:14.341980 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:14.341983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:14.342521 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:14.342704 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:14.342709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:14.343329 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:14.343330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:14.344099 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:14.344101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:14.344421 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:14.344422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:14.344520 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:14.345034 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:14.345051 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:14.346526 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:14.347888 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:14.347894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:14.347896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:14.348107 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:14.348290 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:14.348291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:14.348291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:14.348419 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:14.348658 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:14.348665 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:14.348777 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 22:40:14.348860 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 22:40:14.348971 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260926 22:40:14.349618 14575 heartbeater.cc:347] Connected to a master server at 127.11.107.254:41449
I20260926 22:40:14.349706 14575 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:14.349892 14575 heartbeater.cc:511] Master 127.11.107.254:41449 requested a full tablet report, sending...
I20260926 22:40:14.350051 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 14446
I20260926 22:40:14.350175 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:14.350320 14116 ts_manager.cc:194] Registered new tserver with Master: 35a552a17a5e4191a7e8cd4d4206b239 (127.11.107.195:42561)
I20260926 22:40:14.350384 11695 external_mini_cluster.cc:1428] Setting key 59c16e677ff0dd2c0b64b41f99a06087
I20260926 22:40:14.350701 14116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:50577
I20260926 22:40:14.351646 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 22:40:14.362075 14174 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:14.361265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.11.107.254:41449)
0926 22:40:14.361327 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:14.361328 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:14.361401 (+ 73us) server_negotiation.cc:407] Connection header received
0926 22:40:14.361445 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:14.361446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:14.361454 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:14.361507 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:14.362024 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57740: BlockingRecv error: recv got EOF from 127.0.0.1:57740 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 22:40:14.363073 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 14178
I20260926 22:40:14.367290 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 14312
I20260926 22:40:14.371639 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 14446
I20260926 22:40:14.375715 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 14084
2026-09-26T22:40:14Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1795 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:14 dist-test-slave-j8sq krb5kdc[14587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:14 dist-test-slave-j8sq krb5kdc[14587](info): set up 2 sockets
Sep 26 22:40:14 dist-test-slave-j8sq krb5kdc[14587](info): commencing operation
krb5kdc: starting...
W20260926 22:40:15.410967 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:15 dist-test-slave-j8sq krb5kdc[14587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:40:15.521060 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:44029
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:44029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:15.599115 14600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:15.599336 14600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:15.599377 14600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:15.600513 14600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:15.600577 14600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:15.600602 14600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:15.600632 14600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:15.600665 14600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:15.602171 14600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:44029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:44029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:15.602427 14600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:15.602629 14600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:15.607445 14605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:15.607465 14606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:15.607455 14608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:15.607518 14600 server_base.cc:1034] running on GCE node
I20260926 22:40:15.607903 14600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:15.608178 14600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:15.609339 14600 hybrid_clock.cc:648] HybridClock initialized: now 1790462415609333 us; error 40 us; skew 500 ppm
Sep 26 22:40:15 dist-test-slave-j8sq krb5kdc[14587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462415, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:15.611757 14600 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:40:15.612154 14600 webserver.cc:460] Webserver started at http://127.11.107.254:39887/ using document root <none> and password file <none>
I20260926 22:40:15.612393 14600 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:15.612459 14600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:15.612608 14600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:15.613584 14600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "4aacd4e7ca904bf7adc00acd07782b08"
format_stamp: "Formatted at 2026-09-26 22:40:15 on dist-test-slave-j8sq"
server_key: "b25c8c143b677c4a3c615107815b4de4"
server_key_iv: "82bda12de5c1505e0c5cc43c9efb6b35"
server_key_version: "encryptionkey@0"
I20260926 22:40:15.613994 14600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "4aacd4e7ca904bf7adc00acd07782b08"
format_stamp: "Formatted at 2026-09-26 22:40:15 on dist-test-slave-j8sq"
server_key: "b25c8c143b677c4a3c615107815b4de4"
server_key_iv: "82bda12de5c1505e0c5cc43c9efb6b35"
server_key_version: "encryptionkey@0"
I20260926 22:40:15.615267 14600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:15.616014 14615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:15.616214 14600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:15.616310 14600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "4aacd4e7ca904bf7adc00acd07782b08"
format_stamp: "Formatted at 2026-09-26 22:40:15 on dist-test-slave-j8sq"
server_key: "b25c8c143b677c4a3c615107815b4de4"
server_key_iv: "82bda12de5c1505e0c5cc43c9efb6b35"
server_key_version: "encryptionkey@0"
I20260926 22:40:15.616394 14600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:15.636425 14600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:15.638754 14600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:15.638916 14600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:15.642603 14600 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:44029
I20260926 22:40:15.642635 14667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:44029 every 8 connection(s)
I20260926 22:40:15.643100 14600 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:15.643764 14668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:15.645586 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 14600
I20260926 22:40:15.645709 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:15.645929 11695 external_mini_cluster.cc:1428] Setting key 9876a63e114d5660164b7b2dab7167ce
I20260926 22:40:15.645979 14668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08: Bootstrap starting.
I20260926 22:40:15.646765 14668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:15.647081 14668 log.cc:824] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:15.647799 14668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08: No bootstrap required, opened a new log
I20260926 22:40:15.650115 14668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aacd4e7ca904bf7adc00acd07782b08" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44029 } }
I20260926 22:40:15.650267 14668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:15.650315 14668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aacd4e7ca904bf7adc00acd07782b08, State: Initialized, Role: FOLLOWER
I20260926 22:40:15.650434 14668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aacd4e7ca904bf7adc00acd07782b08" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44029 } }
I20260926 22:40:15.650502 14668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:15.650532 14668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:15.650575 14668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:15.651216 14668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aacd4e7ca904bf7adc00acd07782b08" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44029 } }
I20260926 22:40:15.651300 14668 leader_election.cc:302] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aacd4e7ca904bf7adc00acd07782b08; no voters:
I20260926 22:40:15.651481 14668 leader_election.cc:288] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:15.651512 14673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:15.651614 14673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [term 1 LEADER]: Becoming Leader. State: Replica: 4aacd4e7ca904bf7adc00acd07782b08, State: Running, Role: LEADER
I20260926 22:40:15.651700 14673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aacd4e7ca904bf7adc00acd07782b08" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44029 } }
I20260926 22:40:15.651739 14668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:15.652005 14673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aacd4e7ca904bf7adc00acd07782b08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aacd4e7ca904bf7adc00acd07782b08" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44029 } } }
I20260926 22:40:15.652173 14673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:15.652377 14673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aacd4e7ca904bf7adc00acd07782b08. Latest consensus state: current_term: 1 leader_uuid: "4aacd4e7ca904bf7adc00acd07782b08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aacd4e7ca904bf7adc00acd07782b08" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44029 } } }
I20260926 22:40:15.652500 14673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:15.652797 14680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 22:40:15 dist-test-slave-j8sq krb5kdc[14587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:15.653352 14680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:15.654767 14680 catalog_manager.cc:1347] Generated new cluster ID: 57573573b7614db09abd2cc82ebdd143
I20260926 22:40:15.654822 14680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:15.657311 14671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:15.646406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.11.107.254:44029)
0926 22:40:15.646599 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:15.646608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:15.646641 (+ 33us) server_negotiation.cc:407] Connection header received
0926 22:40:15.647248 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:15.647256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:15.647339 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:15.647454 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:15.648116 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:15.649507 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:15.650292 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:15.650888 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:15.651000 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:15.653996 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:15.654001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:15.654001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:15.654013 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:15.655819 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:15.656201 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:15.656204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:15.656205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:15.656299 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:15.656512 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:15.656513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:15.656514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:15.656746 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:15.656903 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:15.656999 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 22:40:15.657132 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260926 22:40:15.658679 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:40:15.680511 14671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:15.675062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.11.107.254:44029)
0926 22:40:15.675107 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:15.675109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:15.675250 (+ 141us) server_negotiation.cc:407] Connection header received
0926 22:40:15.675381 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:15.675382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:15.675413 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:15.675442 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:15.676037 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:15.677098 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:15.678458 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:15.678857 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:15.678893 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:15.679812 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:15.679813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:15.679814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:15.680117 (+ 303us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:15.680157 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:15.680164 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:15.680195 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:15.680351 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 22:40:15.680409 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:40:15.691634 14680 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:15.692166 14680 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:15.695341 14680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aacd4e7ca904bf7adc00acd07782b08: Generated new TSK 0
I20260926 22:40:15.695528 14680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:40:15.698994 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 14600
2026-09-26T22:40:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1320 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:15 dist-test-slave-j8sq krb5kdc[14701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:15 dist-test-slave-j8sq krb5kdc[14701](info): set up 2 sockets
Sep 26 22:40:15 dist-test-slave-j8sq krb5kdc[14701](info): commencing operation
krb5kdc: starting...
W20260926 22:40:16.738725 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:16 dist-test-slave-j8sq krb5kdc[14701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:40:16.852129 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:34951
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:34951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:16.935158 14714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:16.935382 14714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:16.935418 14714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:16.936609 14714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:16.936677 14714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:16.936704 14714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:16.936736 14714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:16.936759 14714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:16.938225 14714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:34951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:34951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:16.938486 14714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:16.938711 14714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:16.943346 14722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:16.943358 14720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:16.943543 14714 server_base.cc:1034] running on GCE node
W20260926 22:40:16.943389 14719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:16.943876 14714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:16.944169 14714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:16.945330 14714 hybrid_clock.cc:648] HybridClock initialized: now 1790462416945317 us; error 26 us; skew 500 ppm
Sep 26 22:40:16 dist-test-slave-j8sq krb5kdc[14701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462416, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:16.947675 14714 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:40:16.948122 14714 webserver.cc:460] Webserver started at http://127.11.107.254:33181/ using document root <none> and password file <none>
I20260926 22:40:16.948346 14714 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:16.948413 14714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:16.948559 14714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:16.949579 14714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "9055580aba744c5b9f267142fc1d4bc9"
format_stamp: "Formatted at 2026-09-26 22:40:16 on dist-test-slave-j8sq"
server_key: "f7c67b7bb98ef609da4e4ce4ba6304e5"
server_key_iv: "77b2f171512a8a0045f36a5cc38581ca"
server_key_version: "encryptionkey@0"
I20260926 22:40:16.949987 14714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "9055580aba744c5b9f267142fc1d4bc9"
format_stamp: "Formatted at 2026-09-26 22:40:16 on dist-test-slave-j8sq"
server_key: "f7c67b7bb98ef609da4e4ce4ba6304e5"
server_key_iv: "77b2f171512a8a0045f36a5cc38581ca"
server_key_version: "encryptionkey@0"
I20260926 22:40:16.951274 14714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:16.951982 14729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:16.952149 14714 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:16.952260 14714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "9055580aba744c5b9f267142fc1d4bc9"
format_stamp: "Formatted at 2026-09-26 22:40:16 on dist-test-slave-j8sq"
server_key: "f7c67b7bb98ef609da4e4ce4ba6304e5"
server_key_iv: "77b2f171512a8a0045f36a5cc38581ca"
server_key_version: "encryptionkey@0"
I20260926 22:40:16.952340 14714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:16.969154 14714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:16.970921 14714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:16.971081 14714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:16.974929 14714 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:34951
I20260926 22:40:16.974965 14781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:34951 every 8 connection(s)
I20260926 22:40:16.975296 14714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:16.976219 14782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:16.976516 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 14714
I20260926 22:40:16.976630 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:16.976807 11695 external_mini_cluster.cc:1428] Setting key ddec515193a4dc23f06466ce90492ecf
I20260926 22:40:16.979090 14782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9: Bootstrap starting.
I20260926 22:40:16.979661 14782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:16.979960 14782 log.cc:824] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:16.980644 14782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9: No bootstrap required, opened a new log
I20260926 22:40:16.983088 14782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9055580aba744c5b9f267142fc1d4bc9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34951 } }
I20260926 22:40:16.983217 14782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:16.983238 14782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9055580aba744c5b9f267142fc1d4bc9, State: Initialized, Role: FOLLOWER
I20260926 22:40:16.983369 14782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9055580aba744c5b9f267142fc1d4bc9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34951 } }
I20260926 22:40:16.983438 14782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:16.983459 14782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 26 22:40:16 dist-test-slave-j8sq krb5kdc[14701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:16.983523 14782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:16.984186 14782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9055580aba744c5b9f267142fc1d4bc9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34951 } }
I20260926 22:40:16.984314 14782 leader_election.cc:302] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9055580aba744c5b9f267142fc1d4bc9; no voters:
I20260926 22:40:16.984505 14782 leader_election.cc:288] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:16.984541 14787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:16.984704 14787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 9055580aba744c5b9f267142fc1d4bc9, State: Running, Role: LEADER
I20260926 22:40:16.984752 14782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:16.984881 14787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9055580aba744c5b9f267142fc1d4bc9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34951 } }
I20260926 22:40:16.985404 14788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9055580aba744c5b9f267142fc1d4bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9055580aba744c5b9f267142fc1d4bc9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34951 } } }
I20260926 22:40:16.985602 14788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:16.985544 14789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9055580aba744c5b9f267142fc1d4bc9. Latest consensus state: current_term: 1 leader_uuid: "9055580aba744c5b9f267142fc1d4bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9055580aba744c5b9f267142fc1d4bc9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34951 } } }
I20260926 22:40:16.985867 14789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:16.986202 14796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:16.986721 14796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:16.987893 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:16.977334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.11.107.254:34951)
0926 22:40:16.977486 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:16.977501 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:16.977524 (+ 23us) server_negotiation.cc:407] Connection header received
0926 22:40:16.977980 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:16.977986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:16.978034 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:16.978145 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:16.978950 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:16.979867 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:16.980866 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:16.981246 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:16.981326 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:16.984041 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:16.984044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:16.984044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:16.984054 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:16.986219 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:16.986595 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:16.986599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:16.986599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:16.986713 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:16.986951 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:16.986953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:16.986953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:16.987250 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:16.987392 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:16.987536 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 22:40:16.987670 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260926 22:40:16.988590 14796 catalog_manager.cc:1347] Generated new cluster ID: fcfbd9d66e7341c38ec0fd9027df48b0
I20260926 22:40:16.988637 14796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:16.993116 14796 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:16.993796 14796 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:17.007668 14796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9055580aba744c5b9f267142fc1d4bc9: Generated new TSK 0
I20260926 22:40:17.007874 14796 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:40:17.036810 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34951
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:17.116287 14808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:17.116482 14808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:17.116503 14808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:17.117621 14808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:17.117667 14808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:17.117717 14808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:17.119131 14808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:17.119369 14808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:17.119589 14808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:17.124408 14814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:17.124401 14813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:17.124449 14816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:17.124611 14808 server_base.cc:1034] running on GCE node
I20260926 22:40:17.124821 14808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:17.125082 14808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:17.126238 14808 hybrid_clock.cc:648] HybridClock initialized: now 1790462417126239 us; error 44 us; skew 500 ppm
Sep 26 22:40:17 dist-test-slave-j8sq krb5kdc[14701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462417, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:17.128580 14808 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:40:17.128973 14808 webserver.cc:460] Webserver started at http://127.11.107.193:43433/ using document root <none> and password file <none>
I20260926 22:40:17.129201 14808 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:17.129266 14808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:17.129410 14808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:17.130441 14808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "b58b974781a045d19153d672f4b37b65"
format_stamp: "Formatted at 2026-09-26 22:40:17 on dist-test-slave-j8sq"
server_key: "8f95eca633c63cfe3f57f9b283659f47"
server_key_iv: "47923b7b84a219daad79a832ac761e17"
server_key_version: "encryptionkey@0"
I20260926 22:40:17.130785 14808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "b58b974781a045d19153d672f4b37b65"
format_stamp: "Formatted at 2026-09-26 22:40:17 on dist-test-slave-j8sq"
server_key: "8f95eca633c63cfe3f57f9b283659f47"
server_key_iv: "47923b7b84a219daad79a832ac761e17"
server_key_version: "encryptionkey@0"
I20260926 22:40:17.132020 14808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:17.132709 14823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:17.132871 14808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:17.132972 14808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "b58b974781a045d19153d672f4b37b65"
format_stamp: "Formatted at 2026-09-26 22:40:17 on dist-test-slave-j8sq"
server_key: "8f95eca633c63cfe3f57f9b283659f47"
server_key_iv: "47923b7b84a219daad79a832ac761e17"
server_key_version: "encryptionkey@0"
I20260926 22:40:17.133042 14808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:17.140283 14808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:17.142206 14808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:17.142387 14808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:17.142612 14808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:17.142913 14808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:17.142968 14808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:17.143038 14808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:17.143064 14808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:17.149221 14808 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:42635
I20260926 22:40:17.149259 14936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:42635 every 8 connection(s)
I20260926 22:40:17.149592 14808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:17.151196 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 14808
I20260926 22:40:17.151335 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:17.151571 11695 external_mini_cluster.cc:1428] Setting key a5bfc68c19ec16d4157dd398a94fb56d
Sep 26 22:40:17 dist-test-slave-j8sq krb5kdc[14701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462417, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:17.159861 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:17.150291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:57899 (local address 127.11.107.254:34951)
0926 22:40:17.150357 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:17.150359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:17.150561 (+ 202us) server_negotiation.cc:407] Connection header received
0926 22:40:17.151004 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:17.151005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:17.151016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:17.151073 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:17.152292 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.152817 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.154178 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.154516 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.154595 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:17.156998 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:17.157000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:17.157000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:17.157007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:17.158452 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:17.158915 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:17.158916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:17.158916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:17.158963 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:17.159235 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:17.159236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:17.159236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:17.159330 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:17.159378 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:17.159726 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 22:40:17.159796 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
I20260926 22:40:17.160087 14939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:17.150279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34951 (local address 127.11.107.193:57899)
0926 22:40:17.150437 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:17.150473 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:17.150921 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:17.151154 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:17.151159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:17.151965 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:17.152164 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:17.152170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.152908 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.152909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:17.154091 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:17.154094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.154606 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.154607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:17.154752 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:17.155194 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:17.155204 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:17.156907 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:17.158533 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:17.158539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:17.158541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:17.158840 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:17.159029 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:17.159030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:17.159030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:17.159167 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:17.159398 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:17.159404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:17.159576 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 22:40:17.159708 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 22:40:17.159864 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260926 22:40:17.160547 14937 heartbeater.cc:347] Connected to a master server at 127.11.107.254:34951
I20260926 22:40:17.160650 14937 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:17.160815 14937 heartbeater.cc:511] Master 127.11.107.254:34951 requested a full tablet report, sending...
I20260926 22:40:17.161324 14746 ts_manager.cc:194] Registered new tserver with Master: b58b974781a045d19153d672f4b37b65 (127.11.107.193:42635)
I20260926 22:40:17.161973 14746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:57899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:40:17.176620 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34951
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:17.254616 14942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:17.254839 14942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:17.254874 14942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:17.255992 14942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:17.256057 14942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:17.256101 14942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:17.257511 14942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:17.257761 14942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:17.258003 14942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:17.262777 14947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:17.262835 14942 server_base.cc:1034] running on GCE node
W20260926 22:40:17.262805 14950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:17.262805 14948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:17.263195 14942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:17.263440 14942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:17.264598 14942 hybrid_clock.cc:648] HybridClock initialized: now 1790462417264594 us; error 35 us; skew 500 ppm
Sep 26 22:40:17 dist-test-slave-j8sq krb5kdc[14701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462417, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:17.266996 14942 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:40:17.267454 14942 webserver.cc:460] Webserver started at http://127.11.107.194:33241/ using document root <none> and password file <none>
I20260926 22:40:17.267675 14942 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:17.267742 14942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:17.267889 14942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:17.268880 14942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "a00e427289964579be1e661935b11cb4"
format_stamp: "Formatted at 2026-09-26 22:40:17 on dist-test-slave-j8sq"
server_key: "5c493aab8f1b1a60c284be48552b27f3"
server_key_iv: "3eafdb3ba95b14aa7955f6f533ea4cde"
server_key_version: "encryptionkey@0"
I20260926 22:40:17.269241 14942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "a00e427289964579be1e661935b11cb4"
format_stamp: "Formatted at 2026-09-26 22:40:17 on dist-test-slave-j8sq"
server_key: "5c493aab8f1b1a60c284be48552b27f3"
server_key_iv: "3eafdb3ba95b14aa7955f6f533ea4cde"
server_key_version: "encryptionkey@0"
I20260926 22:40:17.270617 14942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:17.271282 14957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:17.271443 14942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:17.271538 14942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "a00e427289964579be1e661935b11cb4"
format_stamp: "Formatted at 2026-09-26 22:40:17 on dist-test-slave-j8sq"
server_key: "5c493aab8f1b1a60c284be48552b27f3"
server_key_iv: "3eafdb3ba95b14aa7955f6f533ea4cde"
server_key_version: "encryptionkey@0"
I20260926 22:40:17.271615 14942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:17.279606 14942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:17.281329 14942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:17.281507 14942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:17.281706 14942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:17.282045 14942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:17.282097 14942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:17.282145 14942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:17.282172 14942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:17.287992 14942 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:33959
I20260926 22:40:17.288038 15070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:33959 every 8 connection(s)
I20260926 22:40:17.288326 14942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:40:17.290907 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 14942
I20260926 22:40:17.290992 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:17.291213 11695 external_mini_cluster.cc:1428] Setting key 76631081a531304ae8ae94627f010dd9
Sep 26 22:40:17 dist-test-slave-j8sq krb5kdc[14701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462417, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:17.297977 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:17.288965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:59409 (local address 127.11.107.254:34951)
0926 22:40:17.289007 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:17.289009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:17.289203 (+ 194us) server_negotiation.cc:407] Connection header received
0926 22:40:17.289646 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:17.289647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:17.289656 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:17.289712 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:17.290623 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.291217 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.292480 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.292667 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.292723 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:17.295150 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:17.295152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:17.295152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:17.295162 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:17.296563 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:17.296977 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:17.296978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:17.296979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:17.297034 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:17.297289 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:17.297290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:17.297291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:17.297400 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:17.297471 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:17.297761 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 22:40:17.297889 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:40:17.298135 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:17.288922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34951 (local address 127.11.107.194:59409)
0926 22:40:17.289070 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:17.289109 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:17.289566 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:17.289761 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:17.289764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:17.290361 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:17.290543 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:17.290547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.291331 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.291333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:17.292396 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:17.292399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.292745 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.292746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:17.292885 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:17.293462 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:17.293469 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:17.295045 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:17.296655 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:17.296662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:17.296664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:17.296886 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:17.297102 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:17.297103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:17.297104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:17.297212 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:17.297476 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:17.297482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:17.297613 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 22:40:17.297740 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 22:40:17.297910 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260926 22:40:17.298597 15071 heartbeater.cc:347] Connected to a master server at 127.11.107.254:34951
I20260926 22:40:17.298688 15071 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:17.298871 15071 heartbeater.cc:511] Master 127.11.107.254:34951 requested a full tablet report, sending...
I20260926 22:40:17.299239 14746 ts_manager.cc:194] Registered new tserver with Master: a00e427289964579be1e661935b11cb4 (127.11.107.194:33959)
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
I20260926 22:40:17.299727 14746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:59409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:40:17.314277 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34951
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:17.393224 15076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:17.393406 15076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:17.393428 15076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:17.394613 15076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:17.394654 15076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:17.394701 15076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:17.396090 15076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:17.396317 15076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:17.396523 15076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:40:17.401259 15076 server_base.cc:1034] running on GCE node
W20260926 22:40:17.401221 15081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:17.401232 15084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:17.401229 15082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:17.401655 15076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:17.401909 15076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:17.403049 15076 hybrid_clock.cc:648] HybridClock initialized: now 1790462417403041 us; error 28 us; skew 500 ppm
Sep 26 22:40:17 dist-test-slave-j8sq krb5kdc[14701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462417, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:17.405287 15076 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:40:17.405732 15076 webserver.cc:460] Webserver started at http://127.11.107.195:46599/ using document root <none> and password file <none>
I20260926 22:40:17.406013 15076 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:17.406080 15076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:17.406221 15076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:17.407217 15076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "5c53fe8ed00b49e1b6eca99c821a4b5a"
format_stamp: "Formatted at 2026-09-26 22:40:17 on dist-test-slave-j8sq"
server_key: "c2cda2724e441b8a1fa7da26048c6b94"
server_key_iv: "2fc4b6fef4cda788b70f7f05edcff567"
server_key_version: "encryptionkey@0"
I20260926 22:40:17.407569 15076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "5c53fe8ed00b49e1b6eca99c821a4b5a"
format_stamp: "Formatted at 2026-09-26 22:40:17 on dist-test-slave-j8sq"
server_key: "c2cda2724e441b8a1fa7da26048c6b94"
server_key_iv: "2fc4b6fef4cda788b70f7f05edcff567"
server_key_version: "encryptionkey@0"
I20260926 22:40:17.408824 15076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:40:17.409512 15091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:17.409685 15076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:17.409775 15076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "5c53fe8ed00b49e1b6eca99c821a4b5a"
format_stamp: "Formatted at 2026-09-26 22:40:17 on dist-test-slave-j8sq"
server_key: "c2cda2724e441b8a1fa7da26048c6b94"
server_key_iv: "2fc4b6fef4cda788b70f7f05edcff567"
server_key_version: "encryptionkey@0"
I20260926 22:40:17.409878 15076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:17.420217 15076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:17.421970 15076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:17.422129 15076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:17.422346 15076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:17.422643 15076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:17.422703 15076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:17.422745 15076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:17.422772 15076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:17.428447 15076 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:38095
I20260926 22:40:17.428498 15204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:38095 every 8 connection(s)
I20260926 22:40:17.428797 15076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:40:17 dist-test-slave-j8sq krb5kdc[14701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462417, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:17.437288 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:17.429490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:54531 (local address 127.11.107.254:34951)
0926 22:40:17.429550 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:17.429551 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:17.429742 (+ 191us) server_negotiation.cc:407] Connection header received
0926 22:40:17.430213 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:17.430214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:17.430224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:17.430277 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:17.431118 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.431597 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.432534 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.432725 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.432776 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:17.434731 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:17.434732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:17.434733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:17.434739 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:17.435891 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:17.436363 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:17.436365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:17.436365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:17.436425 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:17.436672 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:17.436673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:17.436674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:17.436785 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:17.436862 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:17.437119 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 22:40:17.437195 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:40:17.437474 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:17.429468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34951 (local address 127.11.107.195:54531)
0926 22:40:17.429611 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:17.429650 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:17.430130 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:17.430328 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:17.430331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:17.430863 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:17.431046 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:17.431050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.431681 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.431682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:17.432459 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:17.432461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.432797 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.432797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:17.432891 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:17.433299 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:17.433307 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:17.434627 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:17.435988 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:17.435994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:17.435996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:17.436287 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:17.436493 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:17.436494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:17.436494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:17.436600 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:17.436852 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:17.436858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:17.436991 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 22:40:17.437098 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 22:40:17.437255 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260926 22:40:17.437960 15205 heartbeater.cc:347] Connected to a master server at 127.11.107.254:34951
I20260926 22:40:17.438064 15205 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:17.438249 15205 heartbeater.cc:511] Master 127.11.107.254:34951 requested a full tablet report, sending...
I20260926 22:40:17.438393 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 15076
I20260926 22:40:17.438504 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:17.438671 14746 ts_manager.cc:194] Registered new tserver with Master: 5c53fe8ed00b49e1b6eca99c821a4b5a (127.11.107.195:38095)
I20260926 22:40:17.438716 11695 external_mini_cluster.cc:1428] Setting key e8e78858646e31a0358df00c2ea641be
I20260926 22:40:17.439076 14746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:54531
I20260926 22:40:17.439620 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 22:40:17 dist-test-slave-j8sq krb5kdc[14701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462417, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 22:40:17 dist-test-slave-j8sq krb5kdc[14701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462417, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:17.461295 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:17.454737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48402 (local address 127.11.107.254:34951)
0926 22:40:17.454790 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:17.454792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:17.454883 (+ 91us) server_negotiation.cc:407] Connection header received
0926 22:40:17.454941 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:17.454942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:17.454950 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:17.455003 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:17.455590 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.456078 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.456809 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:17.456979 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:17.457027 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:17.459245 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:17.459247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:17.459247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:17.459252 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:17.460402 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:17.460694 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:17.460695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:17.460695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:17.460740 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:17.460895 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:17.460896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:17.460896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:17.461019 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:17.461091 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:17.461175 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 22:40:17.461232 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260926 22:40:17.461591 14746 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48402
I20260926 22:40:17.462111 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 14808
I20260926 22:40:17.466790 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 14942
I20260926 22:40:17.471097 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 15076
I20260926 22:40:17.475081 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 14714
2026-09-26T22:40:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1777 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:17 dist-test-slave-j8sq krb5kdc[15217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:17 dist-test-slave-j8sq krb5kdc[15217](info): set up 2 sockets
Sep 26 22:40:17 dist-test-slave-j8sq krb5kdc[15217](info): commencing operation
krb5kdc: starting...
W20260926 22:40:18.511410 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:18 dist-test-slave-j8sq krb5kdc[15217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:40:18.625242 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:42329
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:42329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:18.703856 15230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:18.704089 15230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:18.704125 15230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:18.705308 15230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:18.705379 15230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:18.705404 15230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:18.705435 15230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:18.705459 15230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:18.706991 15230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:42329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:18.707260 15230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:18.707476 15230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:18.712297 15238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:18.712296 15236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:18.712384 15230 server_base.cc:1034] running on GCE node
W20260926 22:40:18.712330 15235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:18.712834 15230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:18.713155 15230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:18.714313 15230 hybrid_clock.cc:648] HybridClock initialized: now 1790462418714304 us; error 42 us; skew 500 ppm
Sep 26 22:40:18 dist-test-slave-j8sq krb5kdc[15217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462418, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:18.721777 15230 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:40:18.722236 15230 webserver.cc:460] Webserver started at http://127.11.107.254:42211/ using document root <none> and password file <none>
I20260926 22:40:18.722467 15230 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:18.722533 15230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:18.722685 15230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:18.723737 15230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "882e6b225bdb4fc69e70ab0d2c8b64aa"
format_stamp: "Formatted at 2026-09-26 22:40:18 on dist-test-slave-j8sq"
server_key: "68ff02d37c63bb751c3b97395e589420"
server_key_iv: "73b1d3e7369cda278f4cacfff738a883"
server_key_version: "encryptionkey@0"
I20260926 22:40:18.724107 15230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "882e6b225bdb4fc69e70ab0d2c8b64aa"
format_stamp: "Formatted at 2026-09-26 22:40:18 on dist-test-slave-j8sq"
server_key: "68ff02d37c63bb751c3b97395e589420"
server_key_iv: "73b1d3e7369cda278f4cacfff738a883"
server_key_version: "encryptionkey@0"
I20260926 22:40:18.725417 15230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:40:18.726197 15245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:18.726369 15230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:18.726475 15230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "882e6b225bdb4fc69e70ab0d2c8b64aa"
format_stamp: "Formatted at 2026-09-26 22:40:18 on dist-test-slave-j8sq"
server_key: "68ff02d37c63bb751c3b97395e589420"
server_key_iv: "73b1d3e7369cda278f4cacfff738a883"
server_key_version: "encryptionkey@0"
I20260926 22:40:18.726548 15230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:18.743268 15230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:18.744987 15230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:18.745137 15230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:18.749059 15230 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:42329
I20260926 22:40:18.749092 15297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:42329 every 8 connection(s)
I20260926 22:40:18.749573 15230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:18.750247 15298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:18.752404 15298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa: Bootstrap starting.
I20260926 22:40:18.752928 15298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:18.753158 15298 log.cc:824] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:18.753759 15298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa: No bootstrap required, opened a new log
I20260926 22:40:18.755280 15298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882e6b225bdb4fc69e70ab0d2c8b64aa" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 42329 } }
I20260926 22:40:18.755410 15298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:18.755432 15298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 882e6b225bdb4fc69e70ab0d2c8b64aa, State: Initialized, Role: FOLLOWER
I20260926 22:40:18.755564 15298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882e6b225bdb4fc69e70ab0d2c8b64aa" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 42329 } }
I20260926 22:40:18.755642 15298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:18.755671 15298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:18.755698 15298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:18.756500 15298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882e6b225bdb4fc69e70ab0d2c8b64aa" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 42329 } }
I20260926 22:40:18.756582 15298 leader_election.cc:302] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 882e6b225bdb4fc69e70ab0d2c8b64aa; no voters:
I20260926 22:40:18.756702 15298 leader_election.cc:288] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:18.756778 15301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:18.756965 15298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:18.756951 15301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [term 1 LEADER]: Becoming Leader. State: Replica: 882e6b225bdb4fc69e70ab0d2c8b64aa, State: Running, Role: LEADER
I20260926 22:40:18.757174 15301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882e6b225bdb4fc69e70ab0d2c8b64aa" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 42329 } }
I20260926 22:40:18.757560 15303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 882e6b225bdb4fc69e70ab0d2c8b64aa. Latest consensus state: current_term: 1 leader_uuid: "882e6b225bdb4fc69e70ab0d2c8b64aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882e6b225bdb4fc69e70ab0d2c8b64aa" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 42329 } } }
I20260926 22:40:18.757647 15303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:18.757987 15302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "882e6b225bdb4fc69e70ab0d2c8b64aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882e6b225bdb4fc69e70ab0d2c8b64aa" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 42329 } } }
I20260926 22:40:18.758172 15302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:18.757901 15310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:18.758751 15310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:18.759541 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 15230
I20260926 22:40:18.759641 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:18.759806 11695 external_mini_cluster.cc:1428] Setting key 42d528f95649915f3611bd137472be0a
I20260926 22:40:18.760578 15310 catalog_manager.cc:1347] Generated new cluster ID: 0f61fddee70949c4af0e3c052874095a
I20260926 22:40:18.760622 15310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:18.770738 15310 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:18.771284 15310 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:18.774331 15310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 882e6b225bdb4fc69e70ab0d2c8b64aa: Generated new TSK 0
I20260926 22:40:18.774504 15310 catalog_manager.cc:1514] Initializing in-progress tserver states...
Sep 26 22:40:18 dist-test-slave-j8sq krb5kdc[15217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:18.778751 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:18.760301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.11.107.254:42329)
0926 22:40:18.768468 (+ 8167us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:18.768478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:18.768514 (+ 36us) server_negotiation.cc:407] Connection header received
0926 22:40:18.768626 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:18.768632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:18.768686 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:18.768827 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:18.769605 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:18.770957 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:18.772054 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:18.772661 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:18.772758 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:18.775530 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:18.775534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:18.775534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:18.775543 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:18.777107 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:18.777490 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:18.777492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:18.777493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:18.777585 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:18.777778 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:18.777778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:18.777778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:18.778016 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:18.778123 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:18.778264 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 22:40:18.778368 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8122,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:40:18.806216 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:42329
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:18.905781 15324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:18.906047 15324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:18.906070 15324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:18.907224 15324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:18.907262 15324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:18.907289 15324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:18.908766 15324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:18.909009 15324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:18.909243 15324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:18.915076 15329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:18.915048 15332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:18.915069 15330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:18.915149 15324 server_base.cc:1034] running on GCE node
I20260926 22:40:18.915539 15324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:18.915812 15324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:18.916960 15324 hybrid_clock.cc:648] HybridClock initialized: now 1790462418916947 us; error 32 us; skew 500 ppm
Sep 26 22:40:18 dist-test-slave-j8sq krb5kdc[15217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462418, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:18.919389 15324 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:40:18.919824 15324 webserver.cc:460] Webserver started at http://127.11.107.193:41731/ using document root <none> and password file <none>
I20260926 22:40:18.920009 15324 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:18.920049 15324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:18.920130 15324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:18.921166 15324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "16ba815a011c458d8fe83d03924b4ce2"
format_stamp: "Formatted at 2026-09-26 22:40:18 on dist-test-slave-j8sq"
server_key: "d8f2ad3809a97a12768a47fcfccf5855"
server_key_iv: "c5f011b55a1978d50f4e6ba1f7369c9b"
server_key_version: "encryptionkey@0"
I20260926 22:40:18.921463 15324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "16ba815a011c458d8fe83d03924b4ce2"
format_stamp: "Formatted at 2026-09-26 22:40:18 on dist-test-slave-j8sq"
server_key: "d8f2ad3809a97a12768a47fcfccf5855"
server_key_iv: "c5f011b55a1978d50f4e6ba1f7369c9b"
server_key_version: "encryptionkey@0"
I20260926 22:40:18.922788 15324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:18.923442 15339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:18.923614 15324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:18.923704 15324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "16ba815a011c458d8fe83d03924b4ce2"
format_stamp: "Formatted at 2026-09-26 22:40:18 on dist-test-slave-j8sq"
server_key: "d8f2ad3809a97a12768a47fcfccf5855"
server_key_iv: "c5f011b55a1978d50f4e6ba1f7369c9b"
server_key_version: "encryptionkey@0"
I20260926 22:40:18.923758 15324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:18.938819 15324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:18.941017 15324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:18.941123 15324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:18.941295 15324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:18.941599 15324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:18.941633 15324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:18.941686 15324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:18.941711 15324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:18.947077 15324 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:41815
I20260926 22:40:18.947145 15452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:41815 every 8 connection(s)
I20260926 22:40:18.947467 15324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:18.951316 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 15324
I20260926 22:40:18.951423 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:18.951596 11695 external_mini_cluster.cc:1428] Setting key f2d88712238350385ca06dd6d6e5727f
Sep 26 22:40:18 dist-test-slave-j8sq krb5kdc[15217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462418, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:18.957973 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:18.948287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:51489 (local address 127.11.107.254:42329)
0926 22:40:18.948350 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:18.948351 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:18.948559 (+ 208us) server_negotiation.cc:407] Connection header received
0926 22:40:18.949465 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:18.949465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:18.949474 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:18.949526 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:18.950180 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:18.950659 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:18.951963 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:18.952266 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:18.952353 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:18.954914 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:18.954915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:18.954916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:18.954924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:18.956443 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:18.956917 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:18.956918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:18.956918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:18.957024 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:18.957292 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:18.957293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:18.957294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:18.957397 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:18.957451 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:18.957796 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 22:40:18.957911 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:40:18.958271 15455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:18.948249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:42329 (local address 127.11.107.193:51489)
0926 22:40:18.948434 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:18.948475 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:18.949377 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:18.949592 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:18.949595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:18.949832 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:18.950097 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:18.950102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:18.950740 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:18.950740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:18.951862 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:18.951866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:18.952386 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:18.952387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:18.952533 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:18.953031 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:18.953041 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:18.954826 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:18.956531 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:18.956538 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:18.956540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:18.956837 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:18.957080 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:18.957081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:18.957082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:18.957224 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:18.957468 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:18.957474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:18.957651 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 22:40:18.957777 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 22:40:18.957979 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":117,"threads_started":1}
I20260926 22:40:18.958765 15453 heartbeater.cc:347] Connected to a master server at 127.11.107.254:42329
I20260926 22:40:18.958869 15453 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:18.959050 15453 heartbeater.cc:511] Master 127.11.107.254:42329 requested a full tablet report, sending...
I20260926 22:40:18.959537 15262 ts_manager.cc:194] Registered new tserver with Master: 16ba815a011c458d8fe83d03924b4ce2 (127.11.107.193:41815)
I20260926 22:40:18.960134 15262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:51489
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:40:18.976909 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:42329
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:19.055256 15458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:19.055450 15458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:19.055472 15458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:19.056561 15458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:19.056595 15458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:19.056620 15458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:19.058175 15458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:19.058352 15458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:19.058514 15458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:19.063910 15463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:19.063915 15464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:19.063931 15466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:19.063932 15458 server_base.cc:1034] running on GCE node
I20260926 22:40:19.064329 15458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:19.064618 15458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:19.065781 15458 hybrid_clock.cc:648] HybridClock initialized: now 1790462419065763 us; error 35 us; skew 500 ppm
Sep 26 22:40:19 dist-test-slave-j8sq krb5kdc[15217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462419, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:19.068190 15458 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:40:19.068557 15458 webserver.cc:460] Webserver started at http://127.11.107.194:36061/ using document root <none> and password file <none>
I20260926 22:40:19.068737 15458 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:19.068795 15458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:19.068998 15458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:19.070107 15458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "7a87d0b1727643dc87e707ad9ad24646"
format_stamp: "Formatted at 2026-09-26 22:40:19 on dist-test-slave-j8sq"
server_key: "34dce4370d352375ee75e27bb1d4c444"
server_key_iv: "e19a148a92fe1bd6a049adabe2250c69"
server_key_version: "encryptionkey@0"
I20260926 22:40:19.070462 15458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "7a87d0b1727643dc87e707ad9ad24646"
format_stamp: "Formatted at 2026-09-26 22:40:19 on dist-test-slave-j8sq"
server_key: "34dce4370d352375ee75e27bb1d4c444"
server_key_iv: "e19a148a92fe1bd6a049adabe2250c69"
server_key_version: "encryptionkey@0"
I20260926 22:40:19.071729 15458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:19.072428 15473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:19.072624 15458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:19.072806 15458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "7a87d0b1727643dc87e707ad9ad24646"
format_stamp: "Formatted at 2026-09-26 22:40:19 on dist-test-slave-j8sq"
server_key: "34dce4370d352375ee75e27bb1d4c444"
server_key_iv: "e19a148a92fe1bd6a049adabe2250c69"
server_key_version: "encryptionkey@0"
I20260926 22:40:19.072886 15458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:19.096240 15458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:19.098181 15458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:19.098338 15458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:19.098560 15458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:19.098848 15458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:19.098899 15458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:19.098938 15458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:19.098965 15458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:19.104694 15458 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:38165
I20260926 22:40:19.104741 15586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:38165 every 8 connection(s)
I20260926 22:40:19.105041 15458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:40:19 dist-test-slave-j8sq krb5kdc[15217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462419, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:19.111326 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 15458
I20260926 22:40:19.111446 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:19.111668 11695 external_mini_cluster.cc:1428] Setting key 1ef6ce1d271f095fc45fc8519bfeee6e
I20260926 22:40:19.114705 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:19.105731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:60737 (local address 127.11.107.254:42329)
0926 22:40:19.105790 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:19.105792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:19.106004 (+ 212us) server_negotiation.cc:407] Connection header received
0926 22:40:19.106937 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:19.106939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:19.106965 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:19.107029 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:19.107616 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:19.108113 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:19.109195 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:19.109402 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:19.109484 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:19.111536 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:19.111537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:19.111538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:19.111546 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:19.113334 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:19.113716 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:19.113717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:19.113718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:19.113779 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:19.114102 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:19.114102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:19.114103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:19.114207 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:19.114260 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:19.114573 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 22:40:19.114634 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:40:19.114943 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:19.105700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:42329 (local address 127.11.107.194:60737)
0926 22:40:19.105890 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:19.105925 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:19.106830 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:19.107093 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:19.107097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:19.107339 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:19.107523 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:19.107527 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:19.108206 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:19.108207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:19.109098 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:19.109101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:19.109492 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:19.109492 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:19.109614 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:19.110092 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:19.110100 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:19.111401 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:19.113416 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:19.113421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:19.113422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:19.113642 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:19.113875 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:19.113876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:19.113877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:19.114032 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:19.114272 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:19.114277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:19.114434 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 22:40:19.114543 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 22:40:19.114704 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260926 22:40:19.115378 15587 heartbeater.cc:347] Connected to a master server at 127.11.107.254:42329
I20260926 22:40:19.115481 15587 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:19.115667 15587 heartbeater.cc:511] Master 127.11.107.254:42329 requested a full tablet report, sending...
I20260926 22:40:19.116017 15262 ts_manager.cc:194] Registered new tserver with Master: 7a87d0b1727643dc87e707ad9ad24646 (127.11.107.194:38165)
I20260926 22:40:19.116544 15262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:60737
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:40:19.134852 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:42329
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:19.213392 15592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:19.213593 15592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:19.213615 15592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:19.214704 15592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:19.214740 15592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:19.214764 15592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:19.216141 15592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:19.216305 15592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:19.216472 15592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:19.221297 15600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:19.221345 15598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:19.221309 15597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:19.221344 15592 server_base.cc:1034] running on GCE node
I20260926 22:40:19.221692 15592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:19.222018 15592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:19.223192 15592 hybrid_clock.cc:648] HybridClock initialized: now 1790462419223161 us; error 41 us; skew 500 ppm
Sep 26 22:40:19 dist-test-slave-j8sq krb5kdc[15217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462419, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:19.225674 15592 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:40:19.226132 15592 webserver.cc:460] Webserver started at http://127.11.107.195:45981/ using document root <none> and password file <none>
I20260926 22:40:19.226352 15592 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:19.226418 15592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:19.226614 15592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:19.227587 15592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "c701524fe0574ae6b2701a8e142480ec"
format_stamp: "Formatted at 2026-09-26 22:40:19 on dist-test-slave-j8sq"
server_key: "6e1b2cd59d7720f311023eddbd790ec3"
server_key_iv: "8d0751da2b2146b2a1a97793f0091c64"
server_key_version: "encryptionkey@0"
I20260926 22:40:19.227944 15592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "c701524fe0574ae6b2701a8e142480ec"
format_stamp: "Formatted at 2026-09-26 22:40:19 on dist-test-slave-j8sq"
server_key: "6e1b2cd59d7720f311023eddbd790ec3"
server_key_iv: "8d0751da2b2146b2a1a97793f0091c64"
server_key_version: "encryptionkey@0"
I20260926 22:40:19.229323 15592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:19.230103 15607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:19.230281 15592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:19.230376 15592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "c701524fe0574ae6b2701a8e142480ec"
format_stamp: "Formatted at 2026-09-26 22:40:19 on dist-test-slave-j8sq"
server_key: "6e1b2cd59d7720f311023eddbd790ec3"
server_key_iv: "8d0751da2b2146b2a1a97793f0091c64"
server_key_version: "encryptionkey@0"
I20260926 22:40:19.230453 15592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:19.239349 15592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:19.241460 15592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:19.241614 15592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:19.241895 15592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:19.242195 15592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:19.242249 15592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:19.242287 15592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:19.242313 15592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:19.248072 15592 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:45501
I20260926 22:40:19.248113 15720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:45501 every 8 connection(s)
I20260926 22:40:19.248389 15592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:40:19.249284 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 15592
I20260926 22:40:19.249396 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:19.249591 11695 external_mini_cluster.cc:1428] Setting key 443106ffb75d0ad93b2814f7975324e9
Sep 26 22:40:19 dist-test-slave-j8sq krb5kdc[15217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462419, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:19.257248 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:19.248992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:32983 (local address 127.11.107.254:42329)
0926 22:40:19.249085 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:19.249087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:19.249300 (+ 213us) server_negotiation.cc:407] Connection header received
0926 22:40:19.250476 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:19.250477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:19.250488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:19.250543 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:19.251072 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:19.251559 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:19.252498 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:19.252690 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:19.252752 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:19.254800 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:19.254801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:19.254801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:19.254807 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:19.255948 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:19.256300 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:19.256300 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:19.256301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:19.256350 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:19.256584 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:19.256584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:19.256584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:19.256727 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:19.256794 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:19.257062 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 22:40:19.257139 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 22:40:19.257442 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:19.248989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:42329 (local address 127.11.107.195:32983)
0926 22:40:19.249174 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:19.249219 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:19.250372 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:19.250593 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:19.250596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:19.250830 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:19.250998 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:19.251001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:19.251632 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:19.251632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:19.252409 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:19.252411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:19.252793 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:19.252794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:19.252903 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:19.253314 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:19.253322 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:19.254718 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:19.256018 (+ 1300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:19.256023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:19.256025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:19.256237 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:19.256425 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:19.256425 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:19.256426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:19.256526 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:19.256786 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:19.256792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:19.256929 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 22:40:19.257040 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 22:40:19.257222 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260926 22:40:19.257912 15721 heartbeater.cc:347] Connected to a master server at 127.11.107.254:42329
I20260926 22:40:19.257998 15721 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:19.258183 15721 heartbeater.cc:511] Master 127.11.107.254:42329 requested a full tablet report, sending...
I20260926 22:40:19.258687 15262 ts_manager.cc:194] Registered new tserver with Master: c701524fe0574ae6b2701a8e142480ec (127.11.107.195:45501)
I20260926 22:40:19.259052 15262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:32983
I20260926 22:40:19.261118 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 22:40:19 dist-test-slave-j8sq krb5kdc[15217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 22:40:19 dist-test-slave-j8sq krb5kdc[15217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.193@KRBTEST.COM
I20260926 22:40:19.282821 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:19.275550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48186 (local address 127.11.107.193:41815)
0926 22:40:19.275705 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:19.275712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:19.275724 (+ 12us) server_negotiation.cc:407] Connection header received
0926 22:40:19.275757 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:19.275760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:19.275777 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:19.275861 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:19.276421 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:19.277244 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:19.277987 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:19.278298 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:19.278354 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:19.280376 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:19.280384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:19.280385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:19.280391 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:19.281713 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:19.282062 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:19.282064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:19.282065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:19.282130 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:19.282370 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:19.282371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:19.282372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:19.282506 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:19.282628 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:19.282682 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 22:40:19.282746 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
Sep 26 22:40:19 dist-test-slave-j8sq krb5kdc[15217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:19.290511 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:19.284041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.11.107.254:42329)
0926 22:40:19.284097 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:19.284099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:19.284153 (+ 54us) server_negotiation.cc:407] Connection header received
0926 22:40:19.284246 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:19.284247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:19.284256 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:19.284303 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:19.284771 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:19.285248 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:19.285976 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:19.286170 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:19.286250 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:19.288234 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:19.288236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:19.288236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:19.288243 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:19.289364 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:19.289703 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:19.289704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:19.289705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:19.289762 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:19.289982 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:19.289983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:19.289983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:19.290127 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:19.290228 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:19.290340 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 22:40:19.290413 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 22:40:19.290717 15262 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33136
I20260926 22:40:19.291047 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 15324
I20260926 22:40:19.295645 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 15458
I20260926 22:40:19.299633 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 15592
I20260926 22:40:19.303617 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 15230
2026-09-26T22:40:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1828 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:19 dist-test-slave-j8sq krb5kdc[15733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:19 dist-test-slave-j8sq krb5kdc[15733](info): set up 2 sockets
Sep 26 22:40:19 dist-test-slave-j8sq krb5kdc[15733](info): commencing operation
krb5kdc: starting...
W20260926 22:40:20.322942 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:20 dist-test-slave-j8sq krb5kdc[15733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:40:20.436462 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:32977
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:32977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:20.513664 15746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:20.513913 15746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:20.513952 15746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:20.515084 15746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:20.515162 15746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:20.515188 15746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:20.515219 15746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:20.515242 15746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:20.516665 15746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:32977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:32977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:20.516944 15746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:20.517174 15746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:20.521865 15754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:20.521843 15751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:20.522068 15746 server_base.cc:1034] running on GCE node
W20260926 22:40:20.522135 15752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:20.522364 15746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:20.522658 15746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:20.523830 15746 hybrid_clock.cc:648] HybridClock initialized: now 1790462420523815 us; error 28 us; skew 500 ppm
Sep 26 22:40:20 dist-test-slave-j8sq krb5kdc[15733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462420, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:20.526297 15746 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:40:20.526643 15746 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:20.526714 15746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:20.526855 15746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:20.527905 15746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "f945442ebcdb45f08b8d8ccfd6120f0b"
format_stamp: "Formatted at 2026-09-26 22:40:20 on dist-test-slave-j8sq"
server_key: "e99f7269ed713119d127355f0a02d2ae"
server_key_iv: "adda75b8ecf91a85c803032a5aacd20c"
server_key_version: "encryptionkey@0"
I20260926 22:40:20.528277 15746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "f945442ebcdb45f08b8d8ccfd6120f0b"
format_stamp: "Formatted at 2026-09-26 22:40:20 on dist-test-slave-j8sq"
server_key: "e99f7269ed713119d127355f0a02d2ae"
server_key_iv: "adda75b8ecf91a85c803032a5aacd20c"
server_key_version: "encryptionkey@0"
I20260926 22:40:20.529549 15746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:40:20.530339 15760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:20.530514 15746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:20.530610 15746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "f945442ebcdb45f08b8d8ccfd6120f0b"
format_stamp: "Formatted at 2026-09-26 22:40:20 on dist-test-slave-j8sq"
server_key: "e99f7269ed713119d127355f0a02d2ae"
server_key_iv: "adda75b8ecf91a85c803032a5aacd20c"
server_key_version: "encryptionkey@0"
I20260926 22:40:20.530697 15746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:20.549016 15746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:20.550927 15746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:20.551105 15746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:20.555063 15746 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:32977
I20260926 22:40:20.555119 15812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:32977 every 8 connection(s)
I20260926 22:40:20.555428 15746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:20.556341 15813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:20.558679 15813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b: Bootstrap starting.
I20260926 22:40:20.559265 15813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:20.559549 15813 log.cc:824] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:20.560191 15813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b: No bootstrap required, opened a new log
I20260926 22:40:20.561002 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 15746
I20260926 22:40:20.561118 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:20.561308 11695 external_mini_cluster.cc:1428] Setting key c3b55843c75b1b33fb0d1f752028f884
I20260926 22:40:20.562727 15813 raft_consensus.cc:348] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f945442ebcdb45f08b8d8ccfd6120f0b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32977 } }
I20260926 22:40:20.562889 15813 raft_consensus.cc:374] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:20.562928 15813 raft_consensus.cc:729] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f945442ebcdb45f08b8d8ccfd6120f0b, State: Initialized, Role: FOLLOWER
I20260926 22:40:20.563094 15813 consensus_queue.cc:260] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f945442ebcdb45f08b8d8ccfd6120f0b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32977 } }
I20260926 22:40:20.563194 15813 raft_consensus.cc:388] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:20.563231 15813 raft_consensus.cc:482] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:20.563282 15813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:20.563881 15813 raft_consensus.cc:504] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f945442ebcdb45f08b8d8ccfd6120f0b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32977 } }
I20260926 22:40:20.563964 15813 leader_election.cc:302] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f945442ebcdb45f08b8d8ccfd6120f0b; no voters:
I20260926 22:40:20.564157 15813 leader_election.cc:288] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:20.564177 15818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:20.564282 15818 raft_consensus.cc:686] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [term 1 LEADER]: Becoming Leader. State: Replica: f945442ebcdb45f08b8d8ccfd6120f0b, State: Running, Role: LEADER
I20260926 22:40:20.564360 15818 consensus_queue.cc:237] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f945442ebcdb45f08b8d8ccfd6120f0b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32977 } }
I20260926 22:40:20.564402 15813 sys_catalog.cc:564] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:20.564664 15818 sys_catalog.cc:455] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f945442ebcdb45f08b8d8ccfd6120f0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f945442ebcdb45f08b8d8ccfd6120f0b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32977 } } }
I20260926 22:40:20.564919 15818 sys_catalog.cc:458] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:20.565107 15818 sys_catalog.cc:455] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f945442ebcdb45f08b8d8ccfd6120f0b. Latest consensus state: current_term: 1 leader_uuid: "f945442ebcdb45f08b8d8ccfd6120f0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f945442ebcdb45f08b8d8ccfd6120f0b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32977 } } }
I20260926 22:40:20.565171 15818 sys_catalog.cc:458] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:20.565447 15827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:20.565970 15827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:20.568710 15827 catalog_manager.cc:1347] Generated new cluster ID: fd1b2e798b5c4f029ef5d09094e4576d
I20260926 22:40:20.568761 15827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:40:20 dist-test-slave-j8sq krb5kdc[15733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:20.572710 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:20.561811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.11.107.254:32977)
0926 22:40:20.561980 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:20.561991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:20.562021 (+ 30us) server_negotiation.cc:407] Connection header received
0926 22:40:20.562715 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:20.562721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:20.562766 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:20.562876 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:20.563658 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:20.564928 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:20.566116 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:20.566548 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:20.566667 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:20.569439 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:20.569443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:20.569443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:20.569454 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:20.571092 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:20.571488 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:20.571491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:20.571491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:20.571592 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:20.571824 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:20.571824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:20.571825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:20.572043 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:20.572207 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:20.572407 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 22:40:20.572553 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260926 22:40:20.587389 15827 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:20.587920 15827 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:20.592921 15827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b: Generated new TSK 0
I20260926 22:40:20.593088 15827 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:40:20.620451 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:32977
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:20.698736 15839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:20.698947 15839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:20.698983 15839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:20.700129 15839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:20.700201 15839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:20.700248 15839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:20.701718 15839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:32977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:20.701992 15839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:20.702214 15839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:20.706985 15844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:20.706979 15845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:20.707015 15839 server_base.cc:1034] running on GCE node
W20260926 22:40:20.707007 15847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:20.707394 15839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:20.707630 15839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:20.708802 15839 hybrid_clock.cc:648] HybridClock initialized: now 1790462420708788 us; error 43 us; skew 500 ppm
Sep 26 22:40:20 dist-test-slave-j8sq krb5kdc[15733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462420, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:20.711380 15839 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:40:20.711735 15839 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:20.711783 15839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:20.711871 15839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:20.712836 15839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "3d6525dbc41e4ca69fb776f72e70b344"
format_stamp: "Formatted at 2026-09-26 22:40:20 on dist-test-slave-j8sq"
server_key: "3591d58c9cf2fb849aaa50f5363b24be"
server_key_iv: "407bc0d47df597b1d1c86c5267951af8"
server_key_version: "encryptionkey@0"
I20260926 22:40:20.713124 15839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "3d6525dbc41e4ca69fb776f72e70b344"
format_stamp: "Formatted at 2026-09-26 22:40:20 on dist-test-slave-j8sq"
server_key: "3591d58c9cf2fb849aaa50f5363b24be"
server_key_iv: "407bc0d47df597b1d1c86c5267951af8"
server_key_version: "encryptionkey@0"
I20260926 22:40:20.714473 15839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:40:20.715179 15853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:20.715356 15839 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:20.715437 15839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "3d6525dbc41e4ca69fb776f72e70b344"
format_stamp: "Formatted at 2026-09-26 22:40:20 on dist-test-slave-j8sq"
server_key: "3591d58c9cf2fb849aaa50f5363b24be"
server_key_iv: "407bc0d47df597b1d1c86c5267951af8"
server_key_version: "encryptionkey@0"
I20260926 22:40:20.715508 15839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:20.732844 15839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:20.735170 15839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:20.735324 15839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:20.735541 15839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:20.735831 15839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:20.735863 15839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:20.735883 15839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:20.735927 15839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:20.742573 15839 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:44057
I20260926 22:40:20.742626 15966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:44057 every 8 connection(s)
I20260926 22:40:20.742900 15839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:20.745002 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 15839
I20260926 22:40:20.745121 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:20.745333 11695 external_mini_cluster.cc:1428] Setting key 1fbbffa6b6d8d1aeb0807adf1c110e94
Sep 26 22:40:20 dist-test-slave-j8sq krb5kdc[15733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462420, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:20.752835 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:20.743674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:39513 (local address 127.11.107.254:32977)
0926 22:40:20.743745 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:20.743747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:20.744052 (+ 305us) server_negotiation.cc:407] Connection header received
0926 22:40:20.744540 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:20.744541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:20.744553 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:20.744616 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:20.745800 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:20.746329 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:20.747530 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:20.747760 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:20.747814 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:20.750066 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:20.750067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:20.750067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:20.750096 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:20.751467 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:20.751872 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:20.751873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:20.751874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:20.751936 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:20.752181 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:20.752182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:20.752183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:20.752302 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:20.752380 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:20.752658 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 22:40:20.752748 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 22:40:20.753000 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:20.743659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32977 (local address 127.11.107.193:39513)
0926 22:40:20.743930 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:20.743963 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:20.744439 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:20.744689 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:20.744692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:20.745030 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:20.745693 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:20.745698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:20.746423 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:20.746424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:20.747432 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:20.747434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:20.747839 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:20.747840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:20.747973 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:20.748421 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:20.748431 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:20.749955 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:20.751569 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:20.751581 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:20.751583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:20.751796 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:20.752002 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:20.752003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:20.752004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:20.752115 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:20.752369 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:20.752375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:20.752512 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 22:40:20.752634 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 22:40:20.752781 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
I20260926 22:40:20.753422 15967 heartbeater.cc:347] Connected to a master server at 127.11.107.254:32977
I20260926 22:40:20.753508 15967 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:20.753620 15967 heartbeater.cc:511] Master 127.11.107.254:32977 requested a full tablet report, sending...
I20260926 22:40:20.754132 15777 ts_manager.cc:194] Registered new tserver with Master: 3d6525dbc41e4ca69fb776f72e70b344 (127.11.107.193:44057)
I20260926 22:40:20.754799 15777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:39513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:40:20.769938 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:32977
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:20.848951 15972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:20.849135 15972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:20.849184 15972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:20.850267 15972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:20.850301 15972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:20.850327 15972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:20.851691 15972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:32977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:20.851855 15972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:20.852015 15972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:40:20.856848 15972 server_base.cc:1034] running on GCE node
W20260926 22:40:20.856858 15978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:20.856832 15980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:20.856832 15977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:20.857288 15972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:20.857524 15972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:20.858685 15972 hybrid_clock.cc:648] HybridClock initialized: now 1790462420858681 us; error 34 us; skew 500 ppm
Sep 26 22:40:20 dist-test-slave-j8sq krb5kdc[15733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462420, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:20.861128 15972 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:40:20.861496 15972 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:20.861567 15972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:20.861707 15972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:20.862797 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "f4d2200a3f2b49308b52437cf846d68b"
format_stamp: "Formatted at 2026-09-26 22:40:20 on dist-test-slave-j8sq"
server_key: "b7aecb598d39f87e6c8b1608117a9103"
server_key_iv: "9d25cbd73e4cbe725ec150d7b6c6818d"
server_key_version: "encryptionkey@0"
I20260926 22:40:20.863160 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "f4d2200a3f2b49308b52437cf846d68b"
format_stamp: "Formatted at 2026-09-26 22:40:20 on dist-test-slave-j8sq"
server_key: "b7aecb598d39f87e6c8b1608117a9103"
server_key_iv: "9d25cbd73e4cbe725ec150d7b6c6818d"
server_key_version: "encryptionkey@0"
I20260926 22:40:20.864523 15972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:20.865298 15986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:20.865468 15972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:20.865563 15972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "f4d2200a3f2b49308b52437cf846d68b"
format_stamp: "Formatted at 2026-09-26 22:40:20 on dist-test-slave-j8sq"
server_key: "b7aecb598d39f87e6c8b1608117a9103"
server_key_iv: "9d25cbd73e4cbe725ec150d7b6c6818d"
server_key_version: "encryptionkey@0"
I20260926 22:40:20.865643 15972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:20.877954 15972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:20.880330 15972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:20.880499 15972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:20.880736 15972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:20.881049 15972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:20.881102 15972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:20.881150 15972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:20.881179 15972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:20.887101 15972 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:33427
I20260926 22:40:20.887151 16099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:33427 every 8 connection(s)
I20260926 22:40:20.887437 15972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:40:20 dist-test-slave-j8sq krb5kdc[15733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462420, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:20.894186 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 15972
I20260926 22:40:20.894306 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:20.894521 11695 external_mini_cluster.cc:1428] Setting key 9d84e173a713d25446a13c223b50bb29
I20260926 22:40:20.896577 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:20.888159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:38695 (local address 127.11.107.254:32977)
0926 22:40:20.888230 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:20.888232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:20.888417 (+ 185us) server_negotiation.cc:407] Connection header received
0926 22:40:20.888927 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:20.888928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:20.888941 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:20.888990 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:20.889948 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:20.890453 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:20.891456 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:20.891653 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:20.891748 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:20.893646 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:20.893648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:20.893648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:20.893656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:20.895207 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:20.895699 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:20.895700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:20.895700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:20.895749 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:20.896010 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:20.896011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:20.896011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:20.896101 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:20.896149 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:20.896454 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 22:40:20.896514 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 22:40:20.896873 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:20.888145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32977 (local address 127.11.107.194:38695)
0926 22:40:20.888292 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:20.888333 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:20.888821 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:20.889076 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:20.889086 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:20.889343 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:20.889812 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:20.889817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:20.890560 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:20.890561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:20.891358 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:20.891361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:20.891757 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:20.891758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:20.891879 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:20.892282 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:20.892290 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:20.893547 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:20.895299 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:20.895305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:20.895307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:20.895614 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:20.895810 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:20.895811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:20.895811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:20.895952 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:20.896163 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:20.896168 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:20.896326 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 22:40:20.896438 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 22:40:20.896603 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260926 22:40:20.897338 16100 heartbeater.cc:347] Connected to a master server at 127.11.107.254:32977
I20260926 22:40:20.897444 16100 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:20.897585 16100 heartbeater.cc:511] Master 127.11.107.254:32977 requested a full tablet report, sending...
I20260926 22:40:20.897977 15777 ts_manager.cc:194] Registered new tserver with Master: f4d2200a3f2b49308b52437cf846d68b (127.11.107.194:33427)
I20260926 22:40:20.898355 15777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:38695
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:40:20.918077 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:32977
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:20.996502 16105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:20.996697 16105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:20.996721 16105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:20.998010 16105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:20.998049 16105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:20.998076 16105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:20.999543 16105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:32977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:20.999782 16105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:21.000000 16105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:21.004743 16111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:21.004743 16113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:21.004833 16105 server_base.cc:1034] running on GCE node
W20260926 22:40:21.004774 16110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:21.005236 16105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:21.005484 16105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:21.006646 16105 hybrid_clock.cc:648] HybridClock initialized: now 1790462421006623 us; error 48 us; skew 500 ppm
Sep 26 22:40:21 dist-test-slave-j8sq krb5kdc[15733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462421, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:21.009001 16105 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:40:21.009281 16105 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:21.009323 16105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:21.009410 16105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:21.010434 16105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "879de047edd54466be16e17b79c71172"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "658ca833cd5fbacb34530c35b1f26f2a"
server_key_iv: "1161c9336243decbd8d21f9b2e84aca0"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.010733 16105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "879de047edd54466be16e17b79c71172"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "658ca833cd5fbacb34530c35b1f26f2a"
server_key_iv: "1161c9336243decbd8d21f9b2e84aca0"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.012084 16105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:21.013026 16119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:21.013218 16105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:21.013305 16105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "879de047edd54466be16e17b79c71172"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "658ca833cd5fbacb34530c35b1f26f2a"
server_key_iv: "1161c9336243decbd8d21f9b2e84aca0"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.013375 16105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:21.033897 16105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:21.035840 16105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:21.036005 16105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:21.036237 16105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:21.036561 16105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:21.036618 16105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:21.036667 16105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:21.036695 16105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:21.042511 16105 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:39145
I20260926 22:40:21.042579 16232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:39145 every 8 connection(s)
I20260926 22:40:21.042872 16105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:40:21 dist-test-slave-j8sq krb5kdc[15733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462421, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:21.051403 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.043572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:49887 (local address 127.11.107.254:32977)
0926 22:40:21.043644 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.043646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.043836 (+ 190us) server_negotiation.cc:407] Connection header received
0926 22:40:21.044302 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.044303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.044313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.044359 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:21.045238 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.045712 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.046750 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.046925 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.046976 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.049041 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:21.049043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:21.049044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:21.049050 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:21.050315 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:21.050656 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:21.050657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:21.050657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:21.050714 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:21.050917 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:21.050917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:21.050917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:21.051011 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:21.051060 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.051280 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.051340 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 22:40:21.051553 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.043561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32977 (local address 127.11.107.195:49887)
0926 22:40:21.043720 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.043755 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.044221 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.044418 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.044421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.044679 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:21.045168 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:21.045173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.045808 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.045809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:21.046674 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:21.046677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.046996 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.046997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:21.047095 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.047622 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:21.047630 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:21.048956 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:21.050383 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:21.050389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:21.050390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:21.050591 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:21.050771 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:21.050771 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:21.050772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:21.050863 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:21.051063 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:21.051066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:21.051172 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.051252 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.051360 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260926 22:40:21.052049 16233 heartbeater.cc:347] Connected to a master server at 127.11.107.254:32977
I20260926 22:40:21.052156 16233 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:21.052323 16233 heartbeater.cc:511] Master 127.11.107.254:32977 requested a full tablet report, sending...
I20260926 22:40:21.052431 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 16105
I20260926 22:40:21.052549 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:21.052727 15777 ts_manager.cc:194] Registered new tserver with Master: 879de047edd54466be16e17b79c71172 (127.11.107.195:39145)
I20260926 22:40:21.052781 11695 external_mini_cluster.cc:1428] Setting key 4fa68219e77590e11e79261f9bd84500
I20260926 22:40:21.053234 15777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:49887
I20260926 22:40:21.053736 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:40:21.060374 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.055016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.11.107.254:32977)
0926 22:40:21.055079 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.055080 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.055186 (+ 106us) server_negotiation.cc:407] Connection header received
0926 22:40:21.055227 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.055227 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.055236 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.055282 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:21.055808 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.056248 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.056923 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.057087 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.057134 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.058221 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:21.058222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:21.058222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:21.058227 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:21.059333 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:21.059625 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:21.059626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:21.059626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:21.059672 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:21.059828 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:21.059829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:21.059829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:21.059948 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:21.060019 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.060241 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.060310 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 22:40:21.062194 15777 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:21.063808 15777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:21.070390 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.066136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:33427 (local address 127.0.0.1:35884)
0926 22:40:21.066441 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.066482 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.066574 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.066905 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.066908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.066910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:21.066981 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:21.066983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.068343 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.068344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:21.069324 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:21.069327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.070080 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.070081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:21.070171 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.070188 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.070254 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.070309 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":55,"threads_started":1}
I20260926 22:40:21.070464 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.066373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.11.107.193:44057)
0926 22:40:21.066539 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.066543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.066847 (+ 304us) server_negotiation.cc:407] Connection header received
0926 22:40:21.066954 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.066957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.066973 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.067044 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:21.067148 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.067834 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.069288 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.070127 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.070214 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.070264 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.070333 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.070373 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260926 22:40:21.070636 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.066238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.11.107.194:33427)
0926 22:40:21.066470 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.066474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.066554 (+ 80us) server_negotiation.cc:407] Connection header received
0926 22:40:21.066692 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.066694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.066710 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.066804 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:21.067084 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.068240 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.069404 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.069983 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.070079 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.070135 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.070516 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.070565 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
I20260926 22:40:21.070766 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.066370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.11.107.195:39145)
0926 22:40:21.066661 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.066665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.066769 (+ 104us) server_negotiation.cc:407] Connection header received
0926 22:40:21.066812 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.066814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.066831 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.066908 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:21.067096 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.068193 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.069474 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.070155 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.070215 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.070264 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.070615 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.070658 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":81,"threads_started":1}
I20260926 22:40:21.070796 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.066136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:44057 (local address 127.0.0.1:46704)
0926 22:40:21.066597 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.066607 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.066676 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.067041 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.067042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.067044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:21.067091 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:21.067093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.067933 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.067934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:21.069201 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:21.069204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.070212 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.070212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:21.070271 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.070273 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.070322 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.070352 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260926 22:40:21.070917 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.066136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:39145 (local address 127.0.0.1:55972)
0926 22:40:21.066689 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.066699 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.066754 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.066987 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.066988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.066990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:21.067023 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:21.067024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.068296 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.068297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:21.069412 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:21.069414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.070299 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.070302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:21.070362 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.070366 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.070428 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.070852 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":402,"thread_start_us":47,"threads_started":1}
I20260926 22:40:21.071296 16034 tablet_service.cc:1467] Processing CreateTablet for tablet aed8ed292a4648a3a8c9880ce444b799 (DEFAULT_TABLE table=test-table [id=1ccae85a600b46f79e034940b68f063a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:21.071499 15901 tablet_service.cc:1467] Processing CreateTablet for tablet aed8ed292a4648a3a8c9880ce444b799 (DEFAULT_TABLE table=test-table [id=1ccae85a600b46f79e034940b68f063a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:21.071652 16034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed8ed292a4648a3a8c9880ce444b799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:21.071789 15901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed8ed292a4648a3a8c9880ce444b799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:21.075018 16248 tablet_bootstrap.cc:492] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344: Bootstrap starting.
I20260926 22:40:21.075429 16167 tablet_service.cc:1467] Processing CreateTablet for tablet aed8ed292a4648a3a8c9880ce444b799 (DEFAULT_TABLE table=test-table [id=1ccae85a600b46f79e034940b68f063a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:21.075623 16248 tablet_bootstrap.cc:654] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:21.075719 16167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed8ed292a4648a3a8c9880ce444b799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:21.076172 16248 log.cc:824] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:21.076506 16247 tablet_bootstrap.cc:492] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b: Bootstrap starting.
I20260926 22:40:21.076912 16248 tablet_bootstrap.cc:492] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344: No bootstrap required, opened a new log
I20260926 22:40:21.076978 16248 ts_tablet_manager.cc:1397] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:21.077116 16247 tablet_bootstrap.cc:654] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:21.077739 16247 log.cc:824] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:21.078516 16247 tablet_bootstrap.cc:492] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b: No bootstrap required, opened a new log
I20260926 22:40:21.078586 16247 ts_tablet_manager.cc:1397] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:40:21.078609 16251 tablet_bootstrap.cc:492] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172: Bootstrap starting.
I20260926 22:40:21.079136 16251 tablet_bootstrap.cc:654] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:21.079095 16248 raft_consensus.cc:348] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d2200a3f2b49308b52437cf846d68b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33427 } } peers { permanent_uuid: "3d6525dbc41e4ca69fb776f72e70b344" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44057 } } peers { permanent_uuid: "879de047edd54466be16e17b79c71172" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39145 } }
I20260926 22:40:21.079264 16248 raft_consensus.cc:374] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:21.079306 16248 raft_consensus.cc:729] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6525dbc41e4ca69fb776f72e70b344, State: Initialized, Role: FOLLOWER
I20260926 22:40:21.079421 16248 consensus_queue.cc:260] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d2200a3f2b49308b52437cf846d68b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33427 } } peers { permanent_uuid: "3d6525dbc41e4ca69fb776f72e70b344" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44057 } } peers { permanent_uuid: "879de047edd54466be16e17b79c71172" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39145 } }
I20260926 22:40:21.079676 16248 ts_tablet_manager.cc:1428] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:40:21.079708 16251 log.cc:824] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:21.079811 15967 heartbeater.cc:503] Master 127.11.107.254:32977 was elected leader, sending a full tablet report...
I20260926 22:40:21.080456 16251 tablet_bootstrap.cc:492] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172: No bootstrap required, opened a new log
I20260926 22:40:21.080489 16247 raft_consensus.cc:348] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d2200a3f2b49308b52437cf846d68b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33427 } } peers { permanent_uuid: "3d6525dbc41e4ca69fb776f72e70b344" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44057 } } peers { permanent_uuid: "879de047edd54466be16e17b79c71172" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39145 } }
I20260926 22:40:21.080659 16251 ts_tablet_manager.cc:1397] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:40:21.080667 16247 raft_consensus.cc:374] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:21.080706 16247 raft_consensus.cc:729] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d2200a3f2b49308b52437cf846d68b, State: Initialized, Role: FOLLOWER
I20260926 22:40:21.080816 16247 consensus_queue.cc:260] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d2200a3f2b49308b52437cf846d68b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33427 } } peers { permanent_uuid: "3d6525dbc41e4ca69fb776f72e70b344" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44057 } } peers { permanent_uuid: "879de047edd54466be16e17b79c71172" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39145 } }
I20260926 22:40:21.081082 16247 ts_tablet_manager.cc:1428] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:40:21.081101 16100 heartbeater.cc:503] Master 127.11.107.254:32977 was elected leader, sending a full tablet report...
I20260926 22:40:21.082482 16251 raft_consensus.cc:348] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d2200a3f2b49308b52437cf846d68b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33427 } } peers { permanent_uuid: "3d6525dbc41e4ca69fb776f72e70b344" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44057 } } peers { permanent_uuid: "879de047edd54466be16e17b79c71172" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39145 } }
I20260926 22:40:21.082592 16251 raft_consensus.cc:374] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:21.082612 16251 raft_consensus.cc:729] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 879de047edd54466be16e17b79c71172, State: Initialized, Role: FOLLOWER
I20260926 22:40:21.082738 16251 consensus_queue.cc:260] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d2200a3f2b49308b52437cf846d68b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33427 } } peers { permanent_uuid: "3d6525dbc41e4ca69fb776f72e70b344" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44057 } } peers { permanent_uuid: "879de047edd54466be16e17b79c71172" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39145 } }
I20260926 22:40:21.082993 16251 ts_tablet_manager.cc:1428] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:40:21.083037 16233 heartbeater.cc:503] Master 127.11.107.254:32977 was elected leader, sending a full tablet report...
W20260926 22:40:21.138017 16101 tablet.cc:2369] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:40:21.243566 15968 tablet.cc:2369] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:40:21.257532 16254 raft_consensus.cc:482] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:21.257649 16254 raft_consensus.cc:504] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d2200a3f2b49308b52437cf846d68b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33427 } } peers { permanent_uuid: "3d6525dbc41e4ca69fb776f72e70b344" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44057 } } peers { permanent_uuid: "879de047edd54466be16e17b79c71172" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39145 } }
I20260926 22:40:21.258064 16254 leader_election.cc:288] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d6525dbc41e4ca69fb776f72e70b344 (127.11.107.193:44057), 879de047edd54466be16e17b79c71172 (127.11.107.195:39145)
I20260926 22:40:21.260785 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.258385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:37499 (local address 127.11.107.193:44057)
0926 22:40:21.258432 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.258434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.258441 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:21.258461 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.258462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.258468 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.258530 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:21.258705 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.259157 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.260102 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.260446 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.260528 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.260536 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.260687 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.260722 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:40:21.260785 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.258230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:44057 (local address 127.11.107.194:37499)
0926 22:40:21.258285 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.258298 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.258342 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.258547 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.258548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.258549 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:21.258620 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:21.258622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.259238 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.259239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:21.259998 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:21.260000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.260523 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.260524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:21.260607 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.260610 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.260679 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.260716 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":18}
I20260926 22:40:21.260912 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.258249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:39145 (local address 127.11.107.194:34435)
0926 22:40:21.258436 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.258445 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.258520 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.258673 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.258673 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.258675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:21.258746 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:21.258747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.259349 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.259350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:21.260093 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:21.260094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.260630 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.260631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:21.260713 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.260719 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.260796 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.260833 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":120,"threads_started":1}
I20260926 22:40:21.260900 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.258317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:34435 (local address 127.11.107.195:39145)
0926 22:40:21.258380 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.258382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.258505 (+ 123us) server_negotiation.cc:407] Connection header received
0926 22:40:21.258573 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.258574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.258585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.258646 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:21.258812 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.259268 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.260185 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.260549 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.260631 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.260639 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.260805 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.260838 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:40:21.261117 15921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed8ed292a4648a3a8c9880ce444b799" candidate_uuid: "f4d2200a3f2b49308b52437cf846d68b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6525dbc41e4ca69fb776f72e70b344" is_pre_election: true
I20260926 22:40:21.261189 16187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed8ed292a4648a3a8c9880ce444b799" candidate_uuid: "f4d2200a3f2b49308b52437cf846d68b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "879de047edd54466be16e17b79c71172" is_pre_election: true
I20260926 22:40:21.261300 15921 raft_consensus.cc:2454] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d2200a3f2b49308b52437cf846d68b in term 0.
I20260926 22:40:21.261322 16187 raft_consensus.cc:2454] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d2200a3f2b49308b52437cf846d68b in term 0.
I20260926 22:40:21.261533 15990 leader_election.cc:302] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 879de047edd54466be16e17b79c71172, f4d2200a3f2b49308b52437cf846d68b; no voters:
I20260926 22:40:21.261652 16254 raft_consensus.cc:2781] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:21.261704 16254 raft_consensus.cc:482] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:21.261735 16254 raft_consensus.cc:3037] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:21.262409 16254 raft_consensus.cc:504] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d2200a3f2b49308b52437cf846d68b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33427 } } peers { permanent_uuid: "3d6525dbc41e4ca69fb776f72e70b344" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44057 } } peers { permanent_uuid: "879de047edd54466be16e17b79c71172" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39145 } }
I20260926 22:40:21.262539 16254 leader_election.cc:288] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [CANDIDATE]: Term 1 election: Requested vote from peers 3d6525dbc41e4ca69fb776f72e70b344 (127.11.107.193:44057), 879de047edd54466be16e17b79c71172 (127.11.107.195:39145)
I20260926 22:40:21.262691 16187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed8ed292a4648a3a8c9880ce444b799" candidate_uuid: "f4d2200a3f2b49308b52437cf846d68b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "879de047edd54466be16e17b79c71172"
I20260926 22:40:21.262684 15921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed8ed292a4648a3a8c9880ce444b799" candidate_uuid: "f4d2200a3f2b49308b52437cf846d68b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6525dbc41e4ca69fb776f72e70b344"
I20260926 22:40:21.262773 16187 raft_consensus.cc:3037] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:21.262771 15921 raft_consensus.cc:3037] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:21.263456 16187 raft_consensus.cc:2454] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d2200a3f2b49308b52437cf846d68b in term 1.
I20260926 22:40:21.263473 15921 raft_consensus.cc:2454] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d2200a3f2b49308b52437cf846d68b in term 1.
I20260926 22:40:21.263609 15990 leader_election.cc:302] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 879de047edd54466be16e17b79c71172, f4d2200a3f2b49308b52437cf846d68b; no voters:
I20260926 22:40:21.263713 16254 raft_consensus.cc:2781] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:21.263842 16254 raft_consensus.cc:686] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [term 1 LEADER]: Becoming Leader. State: Replica: f4d2200a3f2b49308b52437cf846d68b, State: Running, Role: LEADER
I20260926 22:40:21.263943 16254 consensus_queue.cc:237] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d2200a3f2b49308b52437cf846d68b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33427 } } peers { permanent_uuid: "3d6525dbc41e4ca69fb776f72e70b344" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44057 } } peers { permanent_uuid: "879de047edd54466be16e17b79c71172" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39145 } }
I20260926 22:40:21.264835 15776 catalog_manager.cc:5495] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b reported cstate change: term changed from 0 to 1, leader changed from <none> to f4d2200a3f2b49308b52437cf846d68b (127.11.107.194). New cstate: current_term: 1 leader_uuid: "f4d2200a3f2b49308b52437cf846d68b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d2200a3f2b49308b52437cf846d68b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d6525dbc41e4ca69fb776f72e70b344" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "879de047edd54466be16e17b79c71172" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39145 } health_report { overall_health: UNKNOWN } } }
W20260926 22:40:21.293360 16234 tablet.cc:2369] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:40:21.294809 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.292357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.11.107.194:33427)
0926 22:40:21.292415 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.292416 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.292424 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:40:21.292462 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.292462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.292478 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.292539 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:21.292710 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.293175 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.294017 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:21.294208 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:21.294263 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:21.294427 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:21.294520 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:21.294577 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.294645 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.294661 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:40:21.296514 15921 raft_consensus.cc:1261] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344 [term 1 FOLLOWER]: Refusing update from remote peer f4d2200a3f2b49308b52437cf846d68b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:21.296558 16187 raft_consensus.cc:1261] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172 [term 1 FOLLOWER]: Refusing update from remote peer f4d2200a3f2b49308b52437cf846d68b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:21.296861 16257 consensus_queue.cc:1035] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d6525dbc41e4ca69fb776f72e70b344" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:21.297055 16254 consensus_queue.cc:1035] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [LEADER]: Connected to new peer: Peer: permanent_uuid: "879de047edd54466be16e17b79c71172" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:21.299181 16260 mvcc.cc:205] Tried to move back new op lower bound from 7333734077627998208 to 7333734077497815040. Current Snapshot: MvccSnapshot[applied={T|T < 7333734077627998208}]
I20260926 22:40:21.299366 16262 mvcc.cc:205] Tried to move back new op lower bound from 7333734077627998208 to 7333734077497815040. Current Snapshot: MvccSnapshot[applied={T|T < 7333734077627998208}]
I20260926 22:40:21.299845 16261 mvcc.cc:205] Tried to move back new op lower bound from 7333734077627998208 to 7333734077497815040. Current Snapshot: MvccSnapshot[applied={T|T < 7333734077627998208}]
I20260926 22:40:21.301885 15776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:21.301995 15776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:21.302938 15776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f945442ebcdb45f08b8d8ccfd6120f0b: Sending DeleteTablet for 3 replicas of tablet aed8ed292a4648a3a8c9880ce444b799
I20260926 22:40:21.303257 16034 tablet_service.cc:1514] Processing DeleteTablet for tablet aed8ed292a4648a3a8c9880ce444b799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:21 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:35884
I20260926 22:40:21.303292 16167 tablet_service.cc:1514] Processing DeleteTablet for tablet aed8ed292a4648a3a8c9880ce444b799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:21 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:55972
I20260926 22:40:21.303454 16270 tablet_replica.cc:330] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b: stopping tablet replica
I20260926 22:40:21.303496 15901 tablet_service.cc:1514] Processing DeleteTablet for tablet aed8ed292a4648a3a8c9880ce444b799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:21 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:46704
I20260926 22:40:21.303547 16270 raft_consensus.cc:2229] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:21.303680 16270 raft_consensus.cc:2258] T aed8ed292a4648a3a8c9880ce444b799 P f4d2200a3f2b49308b52437cf846d68b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:21.303810 16271 tablet_replica.cc:330] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172: stopping tablet replica
I20260926 22:40:21.303882 16271 raft_consensus.cc:2229] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:21.303885 16272 tablet_replica.cc:330] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344: stopping tablet replica
I20260926 22:40:21.303962 16271 raft_consensus.cc:2258] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:21.303975 16272 raft_consensus.cc:2229] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:21.304069 16272 raft_consensus.cc:2258] T aed8ed292a4648a3a8c9880ce444b799 P 3d6525dbc41e4ca69fb776f72e70b344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:21.304203 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 15839
I20260926 22:40:21.305485 16271 ts_tablet_manager.cc:1905] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:21.307101 16271 ts_tablet_manager.cc:1918] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:21.307161 16271 log.cc:1196] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/aed8ed292a4648a3a8c9880ce444b799
I20260926 22:40:21.307428 16271 ts_tablet_manager.cc:1939] T aed8ed292a4648a3a8c9880ce444b799 P 879de047edd54466be16e17b79c71172: Deleting consensus metadata
I20260926 22:40:21.307825 15764 catalog_manager.cc:4841] TS 879de047edd54466be16e17b79c71172 (127.11.107.195:39145): tablet aed8ed292a4648a3a8c9880ce444b799 (table test-table [id=1ccae85a600b46f79e034940b68f063a]) successfully deleted
W20260926 22:40:21.309398 15762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:44057 (error 108)
I20260926 22:40:21.309867 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.309702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:44057 (local address 127.0.0.1:46710)
0926 22:40:21.309765 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.309810 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:44057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 22:40:21.309886 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 15972
W20260926 22:40:21.309995 15762 catalog_manager.cc:4584] TS 3d6525dbc41e4ca69fb776f72e70b344 (127.11.107.193:44057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aed8ed292a4648a3a8c9880ce444b799: Network error: Client connection negotiation failed: client connection to 127.11.107.193:44057: connect: Connection refused (error 111)
I20260926 22:40:21.315028 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 16105
I20260926 22:40:21.315052 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.314946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:33427 (local address 127.0.0.1:35914)
0926 22:40:21.315002 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.315027 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.194:33427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 22:40:21.319217 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 15746
2026-09-26T22:40:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2015 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T22:40:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:21Z Disabled control of system clock
I20260926 22:40:21.345322 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:34413
--webserver_interface=127.11.107.254
--webserver_port=0
--builtin_ntp_servers=127.11.107.212:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:34413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 22:40:21.424271 16277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:21.424463 16277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:21.424484 16277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:21.425560 16277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:21.425593 16277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:21.425606 16277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:21.425617 16277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:21.425629 16277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:21.426913 16277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:34413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:34413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:21.427099 16277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:21.427323 16277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:21.432097 16285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:21.432053 16283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:21.432070 16282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:21.432111 16277 server_base.cc:1034] running on GCE node
I20260926 22:40:21.432559 16277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:21.432844 16277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:21.434008 16277 hybrid_clock.cc:648] HybridClock initialized: now 1790462421434004 us; error 38 us; skew 500 ppm
I20260926 22:40:21.435237 16277 webserver.cc:460] Webserver started at http://127.11.107.254:37219/ using document root <none> and password file <none>
I20260926 22:40:21.435472 16277 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:21.435583 16277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:21.435793 16277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:21.436826 16277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "6115f7c7e3674484b45fd8e58dc80f8a"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "eee67ece3262df59ce4f836d18631a22"
server_key_iv: "b6b8356b9f785957a544f23ddf38f16b"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.437191 16277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "6115f7c7e3674484b45fd8e58dc80f8a"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "eee67ece3262df59ce4f836d18631a22"
server_key_iv: "b6b8356b9f785957a544f23ddf38f16b"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.438578 16277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:21.439343 16291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:21.439507 16277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:21.439610 16277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "6115f7c7e3674484b45fd8e58dc80f8a"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "eee67ece3262df59ce4f836d18631a22"
server_key_iv: "b6b8356b9f785957a544f23ddf38f16b"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.439685 16277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:21.442940 16277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:21.443192 16277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:21.443336 16277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:21.447140 16277 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:34413
I20260926 22:40:21.447170 16343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:34413 every 8 connection(s)
I20260926 22:40:21.447476 16277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:21.448117 16344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:21.449126 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 16277
I20260926 22:40:21.449222 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:21.449397 11695 external_mini_cluster.cc:1428] Setting key c4cc54e41848f573e465a94732493008
I20260926 22:40:21.451668 16344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a: Bootstrap starting.
I20260926 22:40:21.452042 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.449930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.11.107.254:34413)
0926 22:40:21.450152 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.450161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.450193 (+ 32us) server_negotiation.cc:407] Connection header received
0926 22:40:21.450716 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.450724 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.450786 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.450933 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:21.451562 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:21.451566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:21.451566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:21.451570 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:21.451613 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:21.451625 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:21.451709 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.451797 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.451849 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":63,"threads_started":1}
I20260926 22:40:21.452342 16344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:21.452641 16344 log.cc:824] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:21.453305 16344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a: No bootstrap required, opened a new log
I20260926 22:40:21.454660 16344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6115f7c7e3674484b45fd8e58dc80f8a" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34413 } }
I20260926 22:40:21.454785 16344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:21.454829 16344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6115f7c7e3674484b45fd8e58dc80f8a, State: Initialized, Role: FOLLOWER
I20260926 22:40:21.454952 16344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6115f7c7e3674484b45fd8e58dc80f8a" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34413 } }
I20260926 22:40:21.455019 16344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:21.455060 16344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:21.455109 16344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:21.455597 16344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6115f7c7e3674484b45fd8e58dc80f8a" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34413 } }
I20260926 22:40:21.455704 16344 leader_election.cc:302] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6115f7c7e3674484b45fd8e58dc80f8a; no voters:
I20260926 22:40:21.455873 16344 leader_election.cc:288] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:21.456122 16349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:21.456123 16344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:21.456296 16349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [term 1 LEADER]: Becoming Leader. State: Replica: 6115f7c7e3674484b45fd8e58dc80f8a, State: Running, Role: LEADER
I20260926 22:40:21.456461 16349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6115f7c7e3674484b45fd8e58dc80f8a" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34413 } }
I20260926 22:40:21.456838 16351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6115f7c7e3674484b45fd8e58dc80f8a. Latest consensus state: current_term: 1 leader_uuid: "6115f7c7e3674484b45fd8e58dc80f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6115f7c7e3674484b45fd8e58dc80f8a" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34413 } } }
I20260926 22:40:21.456934 16351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:21.457124 16358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:21.457234 16350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6115f7c7e3674484b45fd8e58dc80f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6115f7c7e3674484b45fd8e58dc80f8a" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34413 } } }
I20260926 22:40:21.457305 16350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:21.457569 16358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:21.459026 16358 catalog_manager.cc:1347] Generated new cluster ID: ad6ed08fb15c4c1db2e0538d38c2c081
I20260926 22:40:21.459067 16358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:21.472939 16358 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:21.473099 16358 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:21.483605 16358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a: Generated new TSK 0
I20260926 22:40:21.483784 16358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:40:21.490967 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34413
--builtin_ntp_servers=127.11.107.212:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 22:40:21.576133 16368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:21.576370 16368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:21.576404 16368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:21.577581 16368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:21.577643 16368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:21.577695 16368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:21.579110 16368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:21.579384 16368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:21.579684 16368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:21.585271 16373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:21.585281 16374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:21.585552 16368 server_base.cc:1034] running on GCE node
W20260926 22:40:21.585595 16376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:21.585800 16368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:21.586095 16368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:21.587265 16368 hybrid_clock.cc:648] HybridClock initialized: now 1790462421587251 us; error 26 us; skew 500 ppm
I20260926 22:40:21.588465 16368 webserver.cc:460] Webserver started at http://127.11.107.193:46821/ using document root <none> and password file <none>
I20260926 22:40:21.588675 16368 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:21.588716 16368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:21.588800 16368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:21.589756 16368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "edaf37cd5acf44b4965758d2e9df4782"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "6c872ef8ef10aef5c8de4c60e786831a"
server_key_iv: "05c59417279b1d837946d4a0bba60b5f"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.590121 16368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "edaf37cd5acf44b4965758d2e9df4782"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "6c872ef8ef10aef5c8de4c60e786831a"
server_key_iv: "05c59417279b1d837946d4a0bba60b5f"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.591298 16368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:21.591952 16382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:21.592114 16368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:21.592196 16368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "edaf37cd5acf44b4965758d2e9df4782"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "6c872ef8ef10aef5c8de4c60e786831a"
server_key_iv: "05c59417279b1d837946d4a0bba60b5f"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.592279 16368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:21.595469 16368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:21.595721 16368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:21.595853 16368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:21.596056 16368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:21.596338 16368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:21.596369 16368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:21.596422 16368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:21.596447 16368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:21.601693 16368 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:45915
I20260926 22:40:21.601770 16495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:45915 every 8 connection(s)
I20260926 22:40:21.602092 16368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:21.604952 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.602778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:45399 (local address 127.11.107.254:34413)
0926 22:40:21.602822 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.602825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.603101 (+ 276us) server_negotiation.cc:407] Connection header received
0926 22:40:21.603517 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.603518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.603523 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.603576 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:21.604794 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:21.604795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:21.604795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:21.604798 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:21.604807 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:21.604810 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:21.604849 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.604895 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.604905 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:40:21.605190 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.602774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34413 (local address 127.11.107.193:45399)
0926 22:40:21.602965 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.602996 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.603420 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.603623 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.603625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.603633 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:21.604671 (+ 1038us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:21.604672 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:21.604699 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:21.604725 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:21.604849 (+ 124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:21.604851 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:21.604853 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.604902 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.604947 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
I20260926 22:40:21.605684 16496 heartbeater.cc:347] Connected to a master server at 127.11.107.254:34413
I20260926 22:40:21.605756 16496 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:21.605932 16496 heartbeater.cc:511] Master 127.11.107.254:34413 requested a full tablet report, sending...
I20260926 22:40:21.605928 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 16368
I20260926 22:40:21.606026 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:21.606211 11695 external_mini_cluster.cc:1428] Setting key 46ad04d2c53a84dfe2f4664acdaca930
I20260926 22:40:21.606772 16307 ts_manager.cc:194] Registered new tserver with Master: edaf37cd5acf44b4965758d2e9df4782 (127.11.107.193:45915)
I20260926 22:40:21.607184 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34413
--builtin_ntp_servers=127.11.107.212:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 22:40:21.687490 16499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:21.687685 16499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:21.687707 16499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:21.689136 16499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:21.689180 16499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:21.689204 16499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:21.690551 16499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:21.690721 16499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:21.690938 16499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:21.695819 16504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:21.695802 16505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:21.695901 16499 server_base.cc:1034] running on GCE node
W20260926 22:40:21.695784 16507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:21.696252 16499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:21.696439 16499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:21.697580 16499 hybrid_clock.cc:648] HybridClock initialized: now 1790462421697556 us; error 40 us; skew 500 ppm
I20260926 22:40:21.698750 16499 webserver.cc:460] Webserver started at http://127.11.107.194:43751/ using document root <none> and password file <none>
I20260926 22:40:21.698972 16499 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:21.699015 16499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:21.699159 16499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:21.700178 16499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "387a157c5de0427889b697e7a580a66a"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "dfbfaf4089776aaa6d34ef6cb3dcbcd0"
server_key_iv: "db27344e63d70b98a75c397a067a7f1d"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.700526 16499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "387a157c5de0427889b697e7a580a66a"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "dfbfaf4089776aaa6d34ef6cb3dcbcd0"
server_key_iv: "db27344e63d70b98a75c397a067a7f1d"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.701766 16499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:40:21.702522 16513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:21.702702 16499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:21.702787 16499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "387a157c5de0427889b697e7a580a66a"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "dfbfaf4089776aaa6d34ef6cb3dcbcd0"
server_key_iv: "db27344e63d70b98a75c397a067a7f1d"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.702854 16499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:21.706033 16499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:21.706280 16499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:21.706410 16499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:21.706617 16499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:21.706903 16499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:21.706934 16499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:21.706986 16499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:21.707016 16499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:21.712532 16499 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:35645
I20260926 22:40:21.712584 16626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:35645 every 8 connection(s)
I20260926 22:40:21.712879 16499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:40:21.715766 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.713539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:36409 (local address 127.11.107.254:34413)
0926 22:40:21.713601 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.713603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.713795 (+ 192us) server_negotiation.cc:407] Connection header received
0926 22:40:21.714295 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.714296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.714302 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.714367 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:21.715506 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:21.715507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:21.715508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:21.715512 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:21.715525 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:21.715529 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:21.715580 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.715683 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.715697 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:40:21.715945 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.713509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34413 (local address 127.11.107.194:36409)
0926 22:40:21.713686 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.713714 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.714191 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.714433 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.714437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.714446 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:21.715363 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:21.715364 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:21.715390 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:21.715415 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:21.715577 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:21.715581 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:21.715584 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.715691 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.715747 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260926 22:40:21.716364 16627 heartbeater.cc:347] Connected to a master server at 127.11.107.254:34413
I20260926 22:40:21.716432 16627 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:21.716584 16627 heartbeater.cc:511] Master 127.11.107.254:34413 requested a full tablet report, sending...
I20260926 22:40:21.716933 16307 ts_manager.cc:194] Registered new tserver with Master: 387a157c5de0427889b697e7a580a66a (127.11.107.194:35645)
I20260926 22:40:21.721475 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 16499
I20260926 22:40:21.721583 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:21.721788 11695 external_mini_cluster.cc:1428] Setting key f595856aa35d4080471ec54699f696fa
I20260926 22:40:21.722841 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34413
--builtin_ntp_servers=127.11.107.212:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 22:40:21.804877 16630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:21.805097 16630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:21.805141 16630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:21.806433 16630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:21.806490 16630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:21.806531 16630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:21.807842 16630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:21.808099 16630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:21.808382 16630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:21.813117 16636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:21.813091 16638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:21.813066 16635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:21.813282 16630 server_base.cc:1034] running on GCE node
I20260926 22:40:21.813484 16630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:21.813670 16630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:21.814805 16630 hybrid_clock.cc:648] HybridClock initialized: now 1790462421814794 us; error 33 us; skew 500 ppm
I20260926 22:40:21.815968 16630 webserver.cc:460] Webserver started at http://127.11.107.195:41555/ using document root <none> and password file <none>
I20260926 22:40:21.816152 16630 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:21.816192 16630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:21.816277 16630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:21.817306 16630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "830cad3b00014ed68065ffb00aed42ba"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "fffdac5544b220a110e03c2fe90cc84e"
server_key_iv: "c94d6327955735f783c6eb93e5e78fc1"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.817589 16630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "830cad3b00014ed68065ffb00aed42ba"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "fffdac5544b220a110e03c2fe90cc84e"
server_key_iv: "c94d6327955735f783c6eb93e5e78fc1"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.818836 16630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:21.819480 16644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:21.819628 16630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:21.819700 16630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "830cad3b00014ed68065ffb00aed42ba"
format_stamp: "Formatted at 2026-09-26 22:40:21 on dist-test-slave-j8sq"
server_key: "fffdac5544b220a110e03c2fe90cc84e"
server_key_iv: "c94d6327955735f783c6eb93e5e78fc1"
server_key_version: "encryptionkey@0"
I20260926 22:40:21.819749 16630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:21.822777 16630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:21.822964 16630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:21.823050 16630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:21.823184 16630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:21.823400 16630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:21.823431 16630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:21.823449 16630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:21.823462 16630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:21.828820 16630 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:35421
I20260926 22:40:21.828862 16757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:35421 every 8 connection(s)
I20260926 22:40:21.829134 16630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:40:21.831648 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.829758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:40707 (local address 127.11.107.254:34413)
0926 22:40:21.829818 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.829820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.830037 (+ 217us) server_negotiation.cc:407] Connection header received
0926 22:40:21.830417 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.830418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.830422 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.830477 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:21.831467 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:21.831468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:21.831469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:21.831472 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:21.831485 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:21.831488 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:21.831537 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.831585 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.831596 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 22:40:21.831864 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.829717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34413 (local address 127.11.107.195:40707)
0926 22:40:21.829904 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.829944 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.830340 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.830522 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.830525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.830533 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:21.831340 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:21.831340 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:21.831365 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:21.831390 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:21.831538 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:21.831540 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:21.831543 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.831591 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.831634 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260926 22:40:21.832297 16758 heartbeater.cc:347] Connected to a master server at 127.11.107.254:34413
I20260926 22:40:21.832389 16758 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:21.832542 16758 heartbeater.cc:511] Master 127.11.107.254:34413 requested a full tablet report, sending...
I20260926 22:40:21.832914 16307 ts_manager.cc:194] Registered new tserver with Master: 830cad3b00014ed68065ffb00aed42ba (127.11.107.195:35421)
I20260926 22:40:21.837316 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 16630
I20260926 22:40:21.837415 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:21.837599 11695 external_mini_cluster.cc:1428] Setting key d5d7867f6e980a8b3aca1605c326e264
I20260926 22:40:21.838517 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:40:21.840600 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.839708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.11.107.254:34413)
0926 22:40:21.839747 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.839750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.839838 (+ 88us) server_negotiation.cc:407] Connection header received
0926 22:40:21.839887 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.839887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.839892 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.839935 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:21.840387 (+ 452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:21.840387 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:21.840388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:21.840391 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:21.840401 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:21.840404 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:21.840447 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.840483 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.840494 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:40:21.841998 16307 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:21.843546 16307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:21.847294 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.845531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35645 (local address 127.0.0.1:33852)
0926 22:40:21.845712 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.845723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.845771 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.846272 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.846277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.846284 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:21.846860 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:21.846860 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:21.846872 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:21.846889 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:21.847119 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:21.847126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:21.847128 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.847205 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.847218 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
I20260926 22:40:21.847393 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.845893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.11.107.193:45915)
0926 22:40:21.846049 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.846053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.846127 (+ 74us) server_negotiation.cc:407] Connection header received
0926 22:40:21.846230 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.846235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.846244 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.846295 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:21.847071 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:21.847078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:21.847078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:21.847082 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:21.847115 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:21.847138 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:21.847200 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.847292 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.847308 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260926 22:40:21.847419 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.845941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.11.107.194:35645)
0926 22:40:21.846113 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.846117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.846126 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:21.846154 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.846156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.846165 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.846485 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:21.846969 (+ 484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:21.846974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:21.846975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:21.846979 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:21.847016 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:21.847036 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:21.847122 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.847331 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.847344 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260926 22:40:21.847506 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.845741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:45915 (local address 127.0.0.1:35172)
0926 22:40:21.846067 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.846078 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.846126 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.846588 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.846589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.846591 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:21.846988 (+ 397us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:21.846989 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:21.846997 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:21.847010 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:21.847197 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:21.847197 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:21.847198 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.847280 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.847293 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":65,"threads_started":1}
I20260926 22:40:21.847929 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.845522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:35421 (local address 127.0.0.1:56008)
0926 22:40:21.845657 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.845697 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.845828 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.846331 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.846332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.846334 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:21.847246 (+ 912us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:21.847246 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:21.847251 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:21.847260 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:21.847821 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:21.847821 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:21.847822 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.847863 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.847876 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":75,"threads_started":1}
I20260926 22:40:21.847983 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.845612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56008 (local address 127.11.107.195:35421)
0926 22:40:21.845904 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.845909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.845920 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:40:21.845949 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.845951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.845960 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.846032 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:21.847338 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:21.847663 (+ 325us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:21.847663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:21.847680 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:21.847714 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:21.847735 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:21.847796 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.847885 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.847899 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":78,"threads_started":1}
I20260926 22:40:21.848052 16561 tablet_service.cc:1467] Processing CreateTablet for tablet e9e488e2c2d141b5b1fe55c3863195f2 (DEFAULT_TABLE table=test-table [id=1984681717e144cb9cb73f4f0aaf0237]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:21.848104 16430 tablet_service.cc:1467] Processing CreateTablet for tablet e9e488e2c2d141b5b1fe55c3863195f2 (DEFAULT_TABLE table=test-table [id=1984681717e144cb9cb73f4f0aaf0237]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:21.848392 16430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e488e2c2d141b5b1fe55c3863195f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:21.848392 16561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e488e2c2d141b5b1fe55c3863195f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:21.848714 16692 tablet_service.cc:1467] Processing CreateTablet for tablet e9e488e2c2d141b5b1fe55c3863195f2 (DEFAULT_TABLE table=test-table [id=1984681717e144cb9cb73f4f0aaf0237]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:21.848949 16692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e488e2c2d141b5b1fe55c3863195f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:21.851218 16772 tablet_bootstrap.cc:492] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba: Bootstrap starting.
I20260926 22:40:21.851979 16772 tablet_bootstrap.cc:654] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:21.852273 16772 log.cc:824] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:21.852629 16774 tablet_bootstrap.cc:492] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a: Bootstrap starting.
I20260926 22:40:21.853073 16772 tablet_bootstrap.cc:492] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba: No bootstrap required, opened a new log
I20260926 22:40:21.853145 16772 ts_tablet_manager.cc:1397] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:21.853267 16774 tablet_bootstrap.cc:654] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:21.853565 16774 log.cc:824] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:21.853940 16773 tablet_bootstrap.cc:492] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782: Bootstrap starting.
I20260926 22:40:21.854358 16774 tablet_bootstrap.cc:492] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a: No bootstrap required, opened a new log
I20260926 22:40:21.854439 16774 ts_tablet_manager.cc:1397] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:21.854578 16773 tablet_bootstrap.cc:654] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:21.854892 16773 log.cc:824] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:21.855728 16773 tablet_bootstrap.cc:492] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782: No bootstrap required, opened a new log
I20260926 22:40:21.855798 16773 ts_tablet_manager.cc:1397] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:21.855949 16772 raft_consensus.cc:348] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830cad3b00014ed68065ffb00aed42ba" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35421 } } peers { permanent_uuid: "387a157c5de0427889b697e7a580a66a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35645 } } peers { permanent_uuid: "edaf37cd5acf44b4965758d2e9df4782" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45915 } }
I20260926 22:40:21.856141 16772 raft_consensus.cc:374] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:21.856202 16772 raft_consensus.cc:729] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830cad3b00014ed68065ffb00aed42ba, State: Initialized, Role: FOLLOWER
I20260926 22:40:21.856354 16772 consensus_queue.cc:260] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830cad3b00014ed68065ffb00aed42ba" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35421 } } peers { permanent_uuid: "387a157c5de0427889b697e7a580a66a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35645 } } peers { permanent_uuid: "edaf37cd5acf44b4965758d2e9df4782" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45915 } }
I20260926 22:40:21.856606 16772 ts_tablet_manager.cc:1428] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 22:40:21.856907 16758 heartbeater.cc:503] Master 127.11.107.254:34413 was elected leader, sending a full tablet report...
I20260926 22:40:21.856966 16774 raft_consensus.cc:348] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830cad3b00014ed68065ffb00aed42ba" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35421 } } peers { permanent_uuid: "387a157c5de0427889b697e7a580a66a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35645 } } peers { permanent_uuid: "edaf37cd5acf44b4965758d2e9df4782" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45915 } }
I20260926 22:40:21.857096 16774 raft_consensus.cc:374] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:21.857132 16774 raft_consensus.cc:729] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387a157c5de0427889b697e7a580a66a, State: Initialized, Role: FOLLOWER
I20260926 22:40:21.857259 16774 consensus_queue.cc:260] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830cad3b00014ed68065ffb00aed42ba" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35421 } } peers { permanent_uuid: "387a157c5de0427889b697e7a580a66a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35645 } } peers { permanent_uuid: "edaf37cd5acf44b4965758d2e9df4782" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45915 } }
I20260926 22:40:21.857528 16774 ts_tablet_manager.cc:1428] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:40:21.857575 16627 heartbeater.cc:503] Master 127.11.107.254:34413 was elected leader, sending a full tablet report...
I20260926 22:40:21.858336 16773 raft_consensus.cc:348] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830cad3b00014ed68065ffb00aed42ba" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35421 } } peers { permanent_uuid: "387a157c5de0427889b697e7a580a66a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35645 } } peers { permanent_uuid: "edaf37cd5acf44b4965758d2e9df4782" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45915 } }
I20260926 22:40:21.858515 16773 raft_consensus.cc:374] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:21.858549 16773 raft_consensus.cc:729] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edaf37cd5acf44b4965758d2e9df4782, State: Initialized, Role: FOLLOWER
I20260926 22:40:21.858669 16773 consensus_queue.cc:260] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830cad3b00014ed68065ffb00aed42ba" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35421 } } peers { permanent_uuid: "387a157c5de0427889b697e7a580a66a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35645 } } peers { permanent_uuid: "edaf37cd5acf44b4965758d2e9df4782" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45915 } }
I20260926 22:40:21.858850 16773 ts_tablet_manager.cc:1428] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:40:21.858907 16496 heartbeater.cc:503] Master 127.11.107.254:34413 was elected leader, sending a full tablet report...
I20260926 22:40:21.896993 16780 raft_consensus.cc:482] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:21.897207 16780 raft_consensus.cc:504] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830cad3b00014ed68065ffb00aed42ba" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35421 } } peers { permanent_uuid: "387a157c5de0427889b697e7a580a66a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35645 } } peers { permanent_uuid: "edaf37cd5acf44b4965758d2e9df4782" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45915 } }
I20260926 22:40:21.897703 16780 leader_election.cc:288] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830cad3b00014ed68065ffb00aed42ba (127.11.107.195:35421), 387a157c5de0427889b697e7a580a66a (127.11.107.194:35645)
I20260926 22:40:21.899238 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.898018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:35805 (local address 127.11.107.194:35645)
0926 22:40:21.898075 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.898077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.898231 (+ 154us) server_negotiation.cc:407] Connection header received
0926 22:40:21.898296 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.898297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.898303 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.898355 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:21.899021 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:21.899023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:21.899023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:21.899029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:21.899044 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:21.899048 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:21.899100 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.899154 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.899169 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:40:21.899250 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.897915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35645 (local address 127.11.107.193:35805)
0926 22:40:21.898173 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.898181 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.898235 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.898358 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.898360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.898363 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:21.898884 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:21.898885 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:21.898893 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:21.898909 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:21.899106 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:21.899106 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:21.899107 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.899156 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.899169 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":96,"threads_started":1}
I20260926 22:40:21.899866 16581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e488e2c2d141b5b1fe55c3863195f2" candidate_uuid: "edaf37cd5acf44b4965758d2e9df4782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387a157c5de0427889b697e7a580a66a" is_pre_election: true
I20260926 22:40:21.900045 16581 raft_consensus.cc:2454] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edaf37cd5acf44b4965758d2e9df4782 in term 0.
I20260926 22:40:21.900295 16385 leader_election.cc:302] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387a157c5de0427889b697e7a580a66a, edaf37cd5acf44b4965758d2e9df4782; no voters:
I20260926 22:40:21.900462 16780 raft_consensus.cc:2781] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:21.900571 16780 raft_consensus.cc:482] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:21.900616 16780 raft_consensus.cc:3037] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:21.901186 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.897881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:35421 (local address 127.11.107.193:55725)
0926 22:40:21.897989 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.898001 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:21.898041 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:21.900382 (+ 2341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:21.900383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:21.900386 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:21.900832 (+ 446us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:21.900832 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:21.900839 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:21.900853 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:21.901078 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:21.901079 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:21.901080 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:40:21.901128 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 22:40:21.901138 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 22:40:21.901216 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.900194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:55725 (local address 127.11.107.195:35421)
0926 22:40:21.900236 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.900238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.900247 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:21.900270 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.900271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.900275 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.900332 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:21.900949 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:21.900951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:21.900951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:21.900956 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:21.900972 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:21.900977 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:21.901054 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.901133 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.901147 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:40:21.901544 16780 raft_consensus.cc:504] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830cad3b00014ed68065ffb00aed42ba" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35421 } } peers { permanent_uuid: "387a157c5de0427889b697e7a580a66a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35645 } } peers { permanent_uuid: "edaf37cd5acf44b4965758d2e9df4782" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45915 } }
I20260926 22:40:21.901583 16712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e488e2c2d141b5b1fe55c3863195f2" candidate_uuid: "edaf37cd5acf44b4965758d2e9df4782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830cad3b00014ed68065ffb00aed42ba" is_pre_election: true
I20260926 22:40:21.901679 16780 leader_election.cc:288] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [CANDIDATE]: Term 1 election: Requested vote from peers 830cad3b00014ed68065ffb00aed42ba (127.11.107.195:35421), 387a157c5de0427889b697e7a580a66a (127.11.107.194:35645)
I20260926 22:40:21.901731 16712 raft_consensus.cc:2454] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edaf37cd5acf44b4965758d2e9df4782 in term 0.
I20260926 22:40:21.901831 16711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e488e2c2d141b5b1fe55c3863195f2" candidate_uuid: "edaf37cd5acf44b4965758d2e9df4782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830cad3b00014ed68065ffb00aed42ba"
I20260926 22:40:21.901933 16581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e488e2c2d141b5b1fe55c3863195f2" candidate_uuid: "edaf37cd5acf44b4965758d2e9df4782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387a157c5de0427889b697e7a580a66a"
I20260926 22:40:21.901942 16711 raft_consensus.cc:3037] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:21.902012 16581 raft_consensus.cc:3037] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:21.902858 16581 raft_consensus.cc:2454] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edaf37cd5acf44b4965758d2e9df4782 in term 1.
I20260926 22:40:21.902910 16711 raft_consensus.cc:2454] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edaf37cd5acf44b4965758d2e9df4782 in term 1.
I20260926 22:40:21.903000 16385 leader_election.cc:302] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387a157c5de0427889b697e7a580a66a, edaf37cd5acf44b4965758d2e9df4782; no voters:
I20260926 22:40:21.903100 16780 raft_consensus.cc:2781] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:21.903239 16780 raft_consensus.cc:686] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [term 1 LEADER]: Becoming Leader. State: Replica: edaf37cd5acf44b4965758d2e9df4782, State: Running, Role: LEADER
I20260926 22:40:21.903327 16780 consensus_queue.cc:237] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830cad3b00014ed68065ffb00aed42ba" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35421 } } peers { permanent_uuid: "387a157c5de0427889b697e7a580a66a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35645 } } peers { permanent_uuid: "edaf37cd5acf44b4965758d2e9df4782" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45915 } }
I20260926 22:40:21.904062 16308 catalog_manager.cc:5495] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 reported cstate change: term changed from 0 to 1, leader changed from <none> to edaf37cd5acf44b4965758d2e9df4782 (127.11.107.193). New cstate: current_term: 1 leader_uuid: "edaf37cd5acf44b4965758d2e9df4782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830cad3b00014ed68065ffb00aed42ba" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "387a157c5de0427889b697e7a580a66a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edaf37cd5acf44b4965758d2e9df4782" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45915 } health_report { overall_health: HEALTHY } } }
I20260926 22:40:21.909452 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.908767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.11.107.193:45915)
0926 22:40:21.908816 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:21.908817 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:21.908824 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:21.908846 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:21.908847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:21.908855 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:21.908903 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:21.909310 (+ 407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:21.909311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:21.909311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:21.909314 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:21.909326 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:21.909329 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:21.909368 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:21.909405 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 22:40:21.909414 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 22:40:21.910974 16711 raft_consensus.cc:1261] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba [term 1 FOLLOWER]: Refusing update from remote peer edaf37cd5acf44b4965758d2e9df4782: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:21.910976 16581 raft_consensus.cc:1261] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a [term 1 FOLLOWER]: Refusing update from remote peer edaf37cd5acf44b4965758d2e9df4782: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:21.911293 16782 consensus_queue.cc:1035] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830cad3b00014ed68065ffb00aed42ba" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:21.911409 16780 consensus_queue.cc:1035] T e9e488e2c2d141b5b1fe55c3863195f2 P edaf37cd5acf44b4965758d2e9df4782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "387a157c5de0427889b697e7a580a66a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:21.913482 16785 mvcc.cc:205] Tried to move back new op lower bound from 7333734080145154048 to 7333734080116559872. Current Snapshot: MvccSnapshot[applied={T|T < 7333734080145154048}]
I20260926 22:40:21.913704 16786 mvcc.cc:205] Tried to move back new op lower bound from 7333734080145154048 to 7333734080116559872. Current Snapshot: MvccSnapshot[applied={T|T < 7333734080145154048}]
I20260926 22:40:21.916078 16308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:21.916165 16308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:21.916865 16308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6115f7c7e3674484b45fd8e58dc80f8a: Sending DeleteTablet for 3 replicas of tablet e9e488e2c2d141b5b1fe55c3863195f2
I20260926 22:40:21.917155 16692 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e488e2c2d141b5b1fe55c3863195f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:21 UTC) from {username='slave'} at 127.0.0.1:56008
I20260926 22:40:21.917212 16430 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e488e2c2d141b5b1fe55c3863195f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:21 UTC) from {username='slave'} at 127.0.0.1:35172
I20260926 22:40:21.917380 16795 tablet_replica.cc:330] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba: stopping tablet replica
I20260926 22:40:21.917423 16561 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e488e2c2d141b5b1fe55c3863195f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:21 UTC) from {username='slave'} at 127.0.0.1:33852
I20260926 22:40:21.917480 16795 raft_consensus.cc:2229] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:21.917586 16795 raft_consensus.cc:2258] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:21.917752 16797 tablet_replica.cc:330] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a: stopping tablet replica
I20260926 22:40:21.917740 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 16368
I20260926 22:40:21.919771 16795 ts_tablet_manager.cc:1905] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:21.921696 16795 ts_tablet_manager.cc:1918] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:21.917827 16797 raft_consensus.cc:2229] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:21.921772 16795 log.cc:1196] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/e9e488e2c2d141b5b1fe55c3863195f2
I20260926 22:40:21.921912 16797 raft_consensus.cc:2258] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:21.922066 16795 ts_tablet_manager.cc:1939] T e9e488e2c2d141b5b1fe55c3863195f2 P 830cad3b00014ed68065ffb00aed42ba: Deleting consensus metadata
I20260926 22:40:21.922448 16797 ts_tablet_manager.cc:1905] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:21.922477 16292 catalog_manager.cc:4841] TS 830cad3b00014ed68065ffb00aed42ba (127.11.107.195:35421): tablet e9e488e2c2d141b5b1fe55c3863195f2 (table test-table [id=1984681717e144cb9cb73f4f0aaf0237]) successfully deleted
W20260926 22:40:21.923655 16294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:45915 (error 108)
I20260926 22:40:21.923980 16797 ts_tablet_manager.cc:1918] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:21.924010 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 16499
I20260926 22:40:21.924057 16797 log.cc:1196] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/e9e488e2c2d141b5b1fe55c3863195f2
I20260926 22:40:21.924078 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.923976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:45915 (local address 127.0.0.1:35188)
0926 22:40:21.924012 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.924047 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 22:40:21.924314 16797 ts_tablet_manager.cc:1939] T e9e488e2c2d141b5b1fe55c3863195f2 P 387a157c5de0427889b697e7a580a66a: Deleting consensus metadata
W20260926 22:40:21.924922 16294 catalog_manager.cc:4584] TS edaf37cd5acf44b4965758d2e9df4782 (127.11.107.193:45915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9e488e2c2d141b5b1fe55c3863195f2: Network error: Client connection negotiation failed: client connection to 127.11.107.193:45915: connect: Connection refused (error 111)
I20260926 22:40:21.928768 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:21.928643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35645 (local address 127.0.0.1:33854)
0926 22:40:21.928698 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:21.928729 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.194:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 22:40:21.929051 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 16630
I20260926 22:40:21.933375 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 16277
2026-09-26T22:40:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (613 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 22:40:21.952723 11695 mini_oidc.cc:152] Starting JWKS server
I20260926 22:40:21.953683 11695 webserver.cc:460] Webserver started at https://127.0.0.1:40145/ using document root <none> and password file <none>
I20260926 22:40:21.953754 11695 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 22:40:21.953938 11695 webserver.cc:460] Webserver started at http://127.0.0.1:43345/ using document root <none> and password file <none>
2026-09-26T22:40:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:21Z Disabled control of system clock
I20260926 22:40:21.960795 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:34485
--webserver_interface=127.11.107.254
--webserver_port=0
--builtin_ntp_servers=127.11.107.212:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:34485
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40145/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/testchainca.cert with env {}
W20260926 22:40:22.039273 16804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:22.039500 16804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:22.039525 16804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:22.040774 16804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:22.040818 16804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:22.040832 16804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:22.040848 16804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:22.040905 16804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 22:40:22.040925 16804 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 22:40:22.040954 16804 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40145/jwks.json
I20260926 22:40:22.042570 16804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:34485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:34485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40145/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:22.042838 16804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:22.043072 16804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:22.047997 16809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:22.047951 16810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:22.048127 16804 server_base.cc:1034] running on GCE node
W20260926 22:40:22.047951 16812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:22.048419 16804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:22.048715 16804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:22.049887 16804 hybrid_clock.cc:648] HybridClock initialized: now 1790462422049877 us; error 31 us; skew 500 ppm
I20260926 22:40:22.051056 16804 webserver.cc:460] Webserver started at http://127.11.107.254:46823/ using document root <none> and password file <none>
I20260926 22:40:22.051313 16804 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:22.051380 16804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:22.051527 16804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:22.052563 16804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "68f827359a674aeead3b0628ddd94cba"
format_stamp: "Formatted at 2026-09-26 22:40:22 on dist-test-slave-j8sq"
server_key: "bcaaee799bb457a33e1433d2ad0448b2"
server_key_iv: "c0bcdb1a6aba84ac4d90e44c4e62a35c"
server_key_version: "encryptionkey@0"
I20260926 22:40:22.052920 16804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "68f827359a674aeead3b0628ddd94cba"
format_stamp: "Formatted at 2026-09-26 22:40:22 on dist-test-slave-j8sq"
server_key: "bcaaee799bb457a33e1433d2ad0448b2"
server_key_iv: "c0bcdb1a6aba84ac4d90e44c4e62a35c"
server_key_version: "encryptionkey@0"
I20260926 22:40:22.054286 16804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:22.055019 16818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:22.055181 16804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:22.055276 16804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "68f827359a674aeead3b0628ddd94cba"
format_stamp: "Formatted at 2026-09-26 22:40:22 on dist-test-slave-j8sq"
server_key: "bcaaee799bb457a33e1433d2ad0448b2"
server_key_iv: "c0bcdb1a6aba84ac4d90e44c4e62a35c"
server_key_version: "encryptionkey@0"
I20260926 22:40:22.055356 16804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:22.071054 16804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:22.071381 16804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:22.071528 16804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:22.075368 16804 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:34485
I20260926 22:40:22.075417 16870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:34485 every 8 connection(s)
I20260926 22:40:22.075732 16804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:22.076570 16871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:22.078820 16871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba: Bootstrap starting.
I20260926 22:40:22.079386 16871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:22.079672 16871 log.cc:824] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:22.080319 16871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba: No bootstrap required, opened a new log
I20260926 22:40:22.081800 16871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f827359a674aeead3b0628ddd94cba" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34485 } }
I20260926 22:40:22.081991 16871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:22.082046 16871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f827359a674aeead3b0628ddd94cba, State: Initialized, Role: FOLLOWER
I20260926 22:40:22.082188 16871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f827359a674aeead3b0628ddd94cba" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34485 } }
I20260926 22:40:22.082245 16871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:22.082297 16871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:22.082343 16871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:22.082852 16871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f827359a674aeead3b0628ddd94cba" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34485 } }
I20260926 22:40:22.082959 16871 leader_election.cc:302] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68f827359a674aeead3b0628ddd94cba; no voters:
I20260926 22:40:22.083123 16871 leader_election.cc:288] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:22.083180 16874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:22.083319 16874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [term 1 LEADER]: Becoming Leader. State: Replica: 68f827359a674aeead3b0628ddd94cba, State: Running, Role: LEADER
I20260926 22:40:22.083369 16871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:22.083413 16874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f827359a674aeead3b0628ddd94cba" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34485 } }
I20260926 22:40:22.083809 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f827359a674aeead3b0628ddd94cba. Latest consensus state: current_term: 1 leader_uuid: "68f827359a674aeead3b0628ddd94cba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f827359a674aeead3b0628ddd94cba" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34485 } } }
I20260926 22:40:22.083904 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:22.083884 16875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68f827359a674aeead3b0628ddd94cba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f827359a674aeead3b0628ddd94cba" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34485 } } }
I20260926 22:40:22.084475 16883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:22.084481 16875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:22.085129 16883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:22.085132 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 16804
I20260926 22:40:22.085249 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:22.085428 11695 external_mini_cluster.cc:1428] Setting key 9680c453b19e7d89143e19f8872e6298
I20260926 22:40:22.086757 16883 catalog_manager.cc:1347] Generated new cluster ID: 82a61fa5d0ef45e095d2f560afd5bef0
I20260926 22:40:22.086824 16883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:22.095207 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:22.085893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.11.107.254:34485)
0926 22:40:22.086091 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:22.086099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:22.090096 (+ 3997us) server_negotiation.cc:407] Connection header received
0926 22:40:22.090226 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:22.090234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:22.090298 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:22.090435 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:22.090658 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.091963 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.093102 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.093705 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.093815 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:22.094524 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:22.094527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:22.094528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:22.094737 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:22.094775 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:22.094788 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:22.094848 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:22.094926 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 22:40:22.094998 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260926 22:40:22.096118 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:40:22.113483 16883 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:22.114030 16883 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:22.119819 16883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68f827359a674aeead3b0628ddd94cba: Generated new TSK 0
I20260926 22:40:22.120007 16883 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 22:40:22.148873 16901 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 22:40:22.149040 16901 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 22:40:22.149124 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:22.126610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.11.107.254:34485)
0926 22:40:22.126655 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:22.126657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:22.126830 (+ 173us) server_negotiation.cc:407] Connection header received
0926 22:40:22.126925 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:22.126926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:22.126937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:22.126987 (+ 50us) server_negotiation.cc:226] Negotiated authn=JWT
0926 22:40:22.127141 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.127585 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.128693 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.128873 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.128921 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:22.129135 (+ 214us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 22:40:22.148859 (+ 19724us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 22:40:22.149038 (+ 179us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58478: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 22:40:22.149178 16893 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58478: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 22:40:22.149471 16900 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:22.126533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34485 (local address 127.0.0.1:58478)
0926 22:40:22.126753 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:22.126766 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:22.126837 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:22.127001 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:22.127001 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:22.127004 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 22:40:22.127078 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:22.127081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.127674 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.127675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:22.128607 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:22.128610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.128942 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.128943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:22.129009 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:22.129029 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 22:40:22.149000 (+ 19971us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 22:40:22.149118 (+ 118us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.107.254:34485: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.107.254:34485: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 22:40:22.149994 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 16804
2026-09-26T22:40:22Z chronyd exiting
I20260926 22:40:22.186044 11695 test_util.cc:175] -----------------------------------------------
I20260926 22:40:22.186117 11695 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462400949426-11695-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (234 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 22:40:22.186919 11695 mini_oidc.cc:152] Starting JWKS server
I20260926 22:40:22.187608 11695 webserver.cc:460] Webserver started at https://127.0.0.1:37139/ using document root <none> and password file <none>
I20260926 22:40:22.187647 11695 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 22:40:22.187834 11695 webserver.cc:460] Webserver started at http://127.0.0.1:35851/ using document root <none> and password file <none>
2026-09-26T22:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:22Z Disabled control of system clock
I20260926 22:40:22.195221 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:43041
--webserver_interface=127.11.107.254
--webserver_port=0
--builtin_ntp_servers=127.11.107.212:37179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:43041
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37139/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/testchainca.cert with env {}
W20260926 22:40:22.273898 16907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:22.274104 16907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:22.274127 16907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:22.275218 16907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:22.275250 16907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:22.275264 16907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:22.275275 16907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:22.275287 16907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 22:40:22.275300 16907 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 22:40:22.275310 16907 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37139/jwks.json
I20260926 22:40:22.276979 16907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:43041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:43041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37139/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:22.277174 16907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:22.277330 16907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:22.282116 16912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:22.282125 16915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:22.282215 16913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:22.282414 16907 server_base.cc:1034] running on GCE node
I20260926 22:40:22.282619 16907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:22.282840 16907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:22.283973 16907 hybrid_clock.cc:648] HybridClock initialized: now 1790462422283958 us; error 32 us; skew 500 ppm
I20260926 22:40:22.285181 16907 webserver.cc:460] Webserver started at http://127.11.107.254:33131/ using document root <none> and password file <none>
I20260926 22:40:22.285410 16907 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:22.285458 16907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:22.285593 16907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:22.286911 16907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "497981b7f1f84ee285acfde6243cb7a8"
format_stamp: "Formatted at 2026-09-26 22:40:22 on dist-test-slave-j8sq"
server_key: "a1a50c2baeb6e082be56619f8046a4f4"
server_key_iv: "6906d88089babf65fc8171eed30fa0b9"
server_key_version: "encryptionkey@0"
I20260926 22:40:22.287279 16907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "497981b7f1f84ee285acfde6243cb7a8"
format_stamp: "Formatted at 2026-09-26 22:40:22 on dist-test-slave-j8sq"
server_key: "a1a50c2baeb6e082be56619f8046a4f4"
server_key_iv: "6906d88089babf65fc8171eed30fa0b9"
server_key_version: "encryptionkey@0"
I20260926 22:40:22.288511 16907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:22.289173 16921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:22.289351 16907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:22.289436 16907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "497981b7f1f84ee285acfde6243cb7a8"
format_stamp: "Formatted at 2026-09-26 22:40:22 on dist-test-slave-j8sq"
server_key: "a1a50c2baeb6e082be56619f8046a4f4"
server_key_iv: "6906d88089babf65fc8171eed30fa0b9"
server_key_version: "encryptionkey@0"
I20260926 22:40:22.289506 16907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:22.298296 16907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:22.298581 16907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:22.298722 16907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:22.302280 16907 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:43041
I20260926 22:40:22.302345 16973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:43041 every 8 connection(s)
I20260926 22:40:22.302654 16907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:22.303282 16974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:22.305905 16974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8: Bootstrap starting.
I20260926 22:40:22.306463 16974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:22.306738 16974 log.cc:824] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:22.307389 16974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8: No bootstrap required, opened a new log
I20260926 22:40:22.308719 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 16907
I20260926 22:40:22.308807 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:22.308984 11695 external_mini_cluster.cc:1428] Setting key 8b8f2601849ccaa8947c4bb5aa6c8ede
I20260926 22:40:22.308954 16974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497981b7f1f84ee285acfde6243cb7a8" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43041 } }
I20260926 22:40:22.309137 16974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:22.309196 16974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 497981b7f1f84ee285acfde6243cb7a8, State: Initialized, Role: FOLLOWER
I20260926 22:40:22.309343 16974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497981b7f1f84ee285acfde6243cb7a8" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43041 } }
I20260926 22:40:22.309430 16974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:22.309466 16974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:22.309518 16974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:22.310231 16974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497981b7f1f84ee285acfde6243cb7a8" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43041 } }
I20260926 22:40:22.310333 16974 leader_election.cc:302] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 497981b7f1f84ee285acfde6243cb7a8; no voters:
I20260926 22:40:22.310472 16974 leader_election.cc:288] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:22.310705 16974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:22.310729 16979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:22.310840 16979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [term 1 LEADER]: Becoming Leader. State: Replica: 497981b7f1f84ee285acfde6243cb7a8, State: Running, Role: LEADER
I20260926 22:40:22.310920 16979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497981b7f1f84ee285acfde6243cb7a8" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43041 } }
I20260926 22:40:22.311247 16980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "497981b7f1f84ee285acfde6243cb7a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497981b7f1f84ee285acfde6243cb7a8" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43041 } } }
I20260926 22:40:22.311326 16980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:22.311457 16980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 497981b7f1f84ee285acfde6243cb7a8. Latest consensus state: current_term: 1 leader_uuid: "497981b7f1f84ee285acfde6243cb7a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497981b7f1f84ee285acfde6243cb7a8" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43041 } } }
I20260926 22:40:22.311520 16980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:22.311655 16988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:22.312275 16988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:22.313737 16988 catalog_manager.cc:1347] Generated new cluster ID: ad0ef7985f45422880530cfdf3fd64a0
I20260926 22:40:22.314144 16988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:22.314980 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:22.309423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44804 (local address 127.11.107.254:43041)
0926 22:40:22.309639 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:22.309648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:22.309692 (+ 44us) server_negotiation.cc:407] Connection header received
0926 22:40:22.309804 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:22.309811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:22.309915 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:22.310081 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:22.310228 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.311681 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.312884 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.313443 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.313566 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:22.314301 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:22.314305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:22.314305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:22.314492 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:22.314529 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:22.314541 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:22.314597 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:22.314719 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 22:40:22.314788 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260926 22:40:22.318269 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:40:22.340914 16988 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:22.341497 16988 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:22.345916 16988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 497981b7f1f84ee285acfde6243cb7a8: Generated new TSK 0
I20260926 22:40:22.346117 16988 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 22:40:22.376026 17004 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 22:40:22.376120 17004 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 22:40:22.376389 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:22.354377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.11.107.254:43041)
0926 22:40:22.354420 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:22.354422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:22.354525 (+ 103us) server_negotiation.cc:407] Connection header received
0926 22:40:22.354607 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:22.354608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:22.354618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:22.354675 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0926 22:40:22.354836 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.355308 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.356425 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.356630 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.356685 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:22.356918 (+ 233us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 22:40:22.376157 (+ 19239us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 22:40:22.376320 (+ 163us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 22:40:22.376456 16978 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 22:40:22.376541 17003 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:22.354288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:43041 (local address 127.0.0.1:44818)
0926 22:40:22.354447 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:22.354459 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:22.354535 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:22.354700 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:22.354701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:22.354703 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 22:40:22.354770 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:22.354773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.355402 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.355403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:22.356334 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:22.356337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.356719 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.356720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:22.356782 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:22.356787 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 22:40:22.376315 (+ 19528us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 22:40:22.376438 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.107.254:43041: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260926 22:40:22.376883 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 16907
2026-09-26T22:40:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 22:40:22.414264 11695 mini_oidc.cc:152] Starting JWKS server
I20260926 22:40:22.414896 11695 webserver.cc:460] Webserver started at https://127.0.0.1:44731/ using document root <none> and password file <none>
I20260926 22:40:22.414937 11695 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 22:40:22.415042 11695 webserver.cc:460] Webserver started at http://127.0.0.1:44403/ using document root <none> and password file <none>
2026-09-26T22:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:22Z Disabled control of system clock
I20260926 22:40:22.422341 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:41265
--webserver_interface=127.11.107.254
--webserver_port=0
--builtin_ntp_servers=127.11.107.212:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:41265
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44731/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 22:40:22.499954 17010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:22.500151 17010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:22.500172 17010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:22.501324 17010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:22.501363 17010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:22.501376 17010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:22.501389 17010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:22.501400 17010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 22:40:22.501412 17010 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 22:40:22.501425 17010 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44731/jwks.json
I20260926 22:40:22.502703 17010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:41265
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:41265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44731/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:22.502872 17010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:22.503075 17010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:22.507586 17018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:22.507557 17016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:22.507727 17010 server_base.cc:1034] running on GCE node
W20260926 22:40:22.507565 17015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:22.507973 17010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:22.508250 17010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:22.509414 17010 hybrid_clock.cc:648] HybridClock initialized: now 1790462422509394 us; error 37 us; skew 500 ppm
I20260926 22:40:22.510717 17010 webserver.cc:460] Webserver started at http://127.11.107.254:38595/ using document root <none> and password file <none>
I20260926 22:40:22.510948 17010 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:22.510993 17010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:22.511139 17010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:22.512147 17010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "185ba7aa141447d89fc2fec89fbf90c7"
format_stamp: "Formatted at 2026-09-26 22:40:22 on dist-test-slave-j8sq"
server_key: "7365a802154d1d839348db20dfb902a4"
server_key_iv: "d10016d4005dc01d74754b1c1ff64348"
server_key_version: "encryptionkey@0"
I20260926 22:40:22.512516 17010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "185ba7aa141447d89fc2fec89fbf90c7"
format_stamp: "Formatted at 2026-09-26 22:40:22 on dist-test-slave-j8sq"
server_key: "7365a802154d1d839348db20dfb902a4"
server_key_iv: "d10016d4005dc01d74754b1c1ff64348"
server_key_version: "encryptionkey@0"
I20260926 22:40:22.513751 17010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:40:22.514515 17024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:22.514672 17010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:22.514750 17010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "185ba7aa141447d89fc2fec89fbf90c7"
format_stamp: "Formatted at 2026-09-26 22:40:22 on dist-test-slave-j8sq"
server_key: "7365a802154d1d839348db20dfb902a4"
server_key_iv: "d10016d4005dc01d74754b1c1ff64348"
server_key_version: "encryptionkey@0"
I20260926 22:40:22.514799 17010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:22.526510 17010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:22.526741 17010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:22.526839 17010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:22.530977 17010 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:41265
I20260926 22:40:22.531028 17076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:41265 every 8 connection(s)
I20260926 22:40:22.531335 17010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:22.531957 17077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:22.534919 17077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7: Bootstrap starting.
I20260926 22:40:22.535486 17077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:22.535770 17077 log.cc:824] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:22.536116 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 17010
I20260926 22:40:22.536243 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:22.536417 11695 external_mini_cluster.cc:1428] Setting key 594f82283f6737a9b962f10af593288e
I20260926 22:40:22.536563 17077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7: No bootstrap required, opened a new log
I20260926 22:40:22.538739 17077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185ba7aa141447d89fc2fec89fbf90c7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41265 } }
I20260926 22:40:22.538893 17077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:22.538938 17077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185ba7aa141447d89fc2fec89fbf90c7, State: Initialized, Role: FOLLOWER
I20260926 22:40:22.539062 17077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185ba7aa141447d89fc2fec89fbf90c7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41265 } }
I20260926 22:40:22.539141 17077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:22.539184 17077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:22.539230 17077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:22.539774 17077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185ba7aa141447d89fc2fec89fbf90c7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41265 } }
I20260926 22:40:22.539880 17077 leader_election.cc:302] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 185ba7aa141447d89fc2fec89fbf90c7; no voters:
I20260926 22:40:22.540032 17077 leader_election.cc:288] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:22.540086 17082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:22.540223 17082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [term 1 LEADER]: Becoming Leader. State: Replica: 185ba7aa141447d89fc2fec89fbf90c7, State: Running, Role: LEADER
I20260926 22:40:22.540272 17077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:22.540304 17082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185ba7aa141447d89fc2fec89fbf90c7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41265 } }
I20260926 22:40:22.540596 17082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 185ba7aa141447d89fc2fec89fbf90c7. Latest consensus state: current_term: 1 leader_uuid: "185ba7aa141447d89fc2fec89fbf90c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185ba7aa141447d89fc2fec89fbf90c7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41265 } } }
I20260926 22:40:22.540673 17082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:22.540601 17083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "185ba7aa141447d89fc2fec89fbf90c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185ba7aa141447d89fc2fec89fbf90c7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41265 } } }
I20260926 22:40:22.540807 17083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:22.540869 17088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:22.541347 17088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:22.543165 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:22.536922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.11.107.254:41265)
0926 22:40:22.537138 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:22.537147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:22.537171 (+ 24us) server_negotiation.cc:407] Connection header received
0926 22:40:22.537881 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:22.537890 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:22.537960 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:22.538073 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:22.538301 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.539683 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.540875 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.541549 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.541684 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:22.542469 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:22.542472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:22.542473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:22.542672 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:22.542718 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:22.542734 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:22.542804 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:22.542892 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 22:40:22.542968 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260926 22:40:22.543346 17088 catalog_manager.cc:1347] Generated new cluster ID: d8678f635b58485aaba5d0207703c2a8
I20260926 22:40:22.543417 17088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:22.546234 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:40:22.553211 17088 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:22.553719 17088 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:22.556272 17088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 185ba7aa141447d89fc2fec89fbf90c7: Generated new TSK 0
I20260926 22:40:22.556461 17088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:40:22.566347 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:22.562564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.11.107.254:41265)
0926 22:40:22.562638 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:22.562642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:22.562680 (+ 38us) server_negotiation.cc:407] Connection header received
0926 22:40:22.562761 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:22.562762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:22.562772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:22.562833 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0926 22:40:22.562965 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.563452 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.564586 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.564761 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.564804 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:22.564980 (+ 176us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 22:40:22.566152 (+ 1172us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 22:40:22.566279 (+ 127us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38502: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 22:40:22.566402 17081 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38502: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 22:40:22.566443 17106 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:22.562470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:41265 (local address 127.0.0.1:38502)
0926 22:40:22.562630 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:22.562640 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:22.562697 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:22.562845 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:22.562846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:22.562848 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 22:40:22.562908 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:22.562910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.563546 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.563547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:22.564501 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:22.564505 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.564828 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.564829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:22.564888 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:22.564893 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 22:40:22.566249 (+ 1356us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 22:40:22.566343 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.107.254:41265: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260926 22:40:22.566731 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 17010
E20260926 22:40:22.580261 17107 webserver.cc:533] Webserver: sslize failed:
E20260926 22:40:22.580322 17107 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T22:40:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (189 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T22:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:22Z Disabled control of system clock
I20260926 22:40:22.611039 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:43791
--webserver_interface=127.11.107.254
--webserver_port=0
--builtin_ntp_servers=127.11.107.212:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:43791
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/testchainca.cert with env {}
W20260926 22:40:22.689316 17111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:22.689514 17111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:22.689534 17111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:22.690739 17111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:22.690780 17111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:22.690793 17111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:22.690832 17111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:22.690862 17111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 22:40:22.690886 17111 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 22:40:22.690910 17111 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 22:40:22.692399 17111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:43791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:22.692642 17111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:22.692857 17111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:22.697333 17117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:22.697341 17116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:22.697341 17119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:22.697587 17111 server_base.cc:1034] running on GCE node
I20260926 22:40:22.697894 17111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:22.698182 17111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:22.699324 17111 hybrid_clock.cc:648] HybridClock initialized: now 1790462422699307 us; error 34 us; skew 500 ppm
I20260926 22:40:22.700429 17111 webserver.cc:460] Webserver started at http://127.11.107.254:35841/ using document root <none> and password file <none>
I20260926 22:40:22.700660 17111 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:22.700712 17111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:22.700851 17111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:22.701977 17111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "df0c7180dff34777ac29f496c94c30d4"
format_stamp: "Formatted at 2026-09-26 22:40:22 on dist-test-slave-j8sq"
server_key: "7c8075c1d1278514a47fb7f296742b60"
server_key_iv: "a39bad5c0469228d79ba1e2b28db9ed0"
server_key_version: "encryptionkey@0"
I20260926 22:40:22.702366 17111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "df0c7180dff34777ac29f496c94c30d4"
format_stamp: "Formatted at 2026-09-26 22:40:22 on dist-test-slave-j8sq"
server_key: "7c8075c1d1278514a47fb7f296742b60"
server_key_iv: "a39bad5c0469228d79ba1e2b28db9ed0"
server_key_version: "encryptionkey@0"
I20260926 22:40:22.703626 17111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:22.704345 17125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:22.704522 17111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:22.704617 17111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "df0c7180dff34777ac29f496c94c30d4"
format_stamp: "Formatted at 2026-09-26 22:40:22 on dist-test-slave-j8sq"
server_key: "7c8075c1d1278514a47fb7f296742b60"
server_key_iv: "a39bad5c0469228d79ba1e2b28db9ed0"
server_key_version: "encryptionkey@0"
I20260926 22:40:22.704708 17111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:22.720659 17111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:22.720999 17111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:22.721153 17111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:22.725179 17111 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:43791
I20260926 22:40:22.725224 17177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:43791 every 8 connection(s)
I20260926 22:40:22.725548 17111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:22.726469 17178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:22.728647 17178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4: Bootstrap starting.
I20260926 22:40:22.729202 17178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:22.729471 17178 log.cc:824] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:22.730161 17178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4: No bootstrap required, opened a new log
I20260926 22:40:22.732208 17178 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0c7180dff34777ac29f496c94c30d4" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43791 } }
I20260926 22:40:22.732350 17178 raft_consensus.cc:374] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:22.732373 17178 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0c7180dff34777ac29f496c94c30d4, State: Initialized, Role: FOLLOWER
I20260926 22:40:22.732470 17178 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0c7180dff34777ac29f496c94c30d4" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43791 } }
I20260926 22:40:22.732517 17178 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:22.732537 17178 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:22.732565 17178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:22.733009 17178 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0c7180dff34777ac29f496c94c30d4" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43791 } }
I20260926 22:40:22.733088 17178 leader_election.cc:302] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0c7180dff34777ac29f496c94c30d4; no voters:
I20260926 22:40:22.733206 17178 leader_election.cc:288] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:22.733263 17181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:22.733421 17181 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [term 1 LEADER]: Becoming Leader. State: Replica: df0c7180dff34777ac29f496c94c30d4, State: Running, Role: LEADER
I20260926 22:40:22.733467 17178 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:22.733513 17181 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0c7180dff34777ac29f496c94c30d4" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43791 } }
I20260926 22:40:22.733901 17182 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df0c7180dff34777ac29f496c94c30d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0c7180dff34777ac29f496c94c30d4" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43791 } } }
I20260926 22:40:22.733999 17182 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:22.733999 17183 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0c7180dff34777ac29f496c94c30d4. Latest consensus state: current_term: 1 leader_uuid: "df0c7180dff34777ac29f496c94c30d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0c7180dff34777ac29f496c94c30d4" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43791 } } }
I20260926 22:40:22.734099 17183 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:22.734297 17190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:22.734923 17190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:22.735311 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 17111
I20260926 22:40:22.735406 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:22.735585 11695 external_mini_cluster.cc:1428] Setting key 56aa5febfb0daf3e8e559dd8bc5e014a
I20260926 22:40:22.736716 17190 catalog_manager.cc:1347] Generated new cluster ID: 5fd08e7f7e5744618abaf69b8b681e6f
I20260926 22:40:22.736773 17190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:22.742666 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:22.736013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.11.107.254:43791)
0926 22:40:22.736259 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:22.736270 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:22.736305 (+ 35us) server_negotiation.cc:407] Connection header received
0926 22:40:22.736500 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:22.736506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:22.736557 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:22.736694 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:22.738146 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.739481 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.740604 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.741184 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.741287 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:22.742005 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:22.742008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:22.742009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:22.742202 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:22.742243 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:22.742256 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:22.742312 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:22.742406 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 22:40:22.742476 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260926 22:40:22.743471 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:40:22.747759 17190 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:22.748251 17190 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:22.751761 17190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df0c7180dff34777ac29f496c94c30d4: Generated new TSK 0
I20260926 22:40:22.751937 17190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:40:22.799973 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:22.759651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.11.107.254:43791)
0926 22:40:22.759695 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:22.759697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:22.759803 (+ 106us) server_negotiation.cc:407] Connection header received
0926 22:40:22.759873 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:22.759874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:22.759884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:22.759942 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0926 22:40:22.760061 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.760536 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.761887 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.762114 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.762173 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:22.762349 (+ 176us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 22:40:22.799690 (+ 37341us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 22:40:22.799894 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50726: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 22:40:22.800051 17200 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50726: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 22:40:22.800031 17207 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:22.759558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:43791 (local address 127.0.0.1:50726)
0926 22:40:22.759726 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:22.759738 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:22.759804 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:22.759943 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:22.759943 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:22.759945 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 22:40:22.760002 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:22.760004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.760812 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.760813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:22.761773 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:22.761775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:22.762185 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:22.762185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:22.762254 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:22.762260 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 22:40:22.799829 (+ 37569us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 22:40:22.799944 (+ 115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.107.254:43791: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":120,"threads_started":1}
I20260926 22:40:22.800390 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 17111
2026-09-26T22:40:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (213 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:23 dist-test-slave-j8sq krb5kdc[17219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:23 dist-test-slave-j8sq krb5kdc[17219](info): set up 2 sockets
Sep 26 22:40:23 dist-test-slave-j8sq krb5kdc[17219](info): commencing operation
krb5kdc: starting...
W20260926 22:40:24.006951 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:40:24.120697 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:37725
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:37725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:24.200234 17232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:24.200502 17232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:24.200541 17232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:24.201885 17232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:24.201949 17232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:24.201975 17232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:24.202004 17232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:24.202034 17232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:24.203462 17232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:37725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:37725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:24.203743 17232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:24.203961 17232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:24.208705 17238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:24.208731 17237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:24.208746 17240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:24.208863 17232 server_base.cc:1034] running on GCE node
I20260926 22:40:24.209065 17232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:24.209396 17232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:24.210561 17232 hybrid_clock.cc:648] HybridClock initialized: now 1790462424210547 us; error 45 us; skew 500 ppm
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:24.216073 17232 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:40:24.216495 17232 webserver.cc:460] Webserver started at http://127.11.107.254:39637/ using document root <none> and password file <none>
I20260926 22:40:24.216733 17232 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:24.216802 17232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:24.216974 17232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:24.218083 17232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "943d8c6c3eb0490788f0a9587353e5b8"
format_stamp: "Formatted at 2026-09-26 22:40:24 on dist-test-slave-j8sq"
server_key: "887ef6ce674b10aaf3dc45ae5f34b8a4"
server_key_iv: "6c22d1746507dff99fa16a0c55bb05bd"
server_key_version: "encryptionkey@0"
I20260926 22:40:24.218453 17232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "943d8c6c3eb0490788f0a9587353e5b8"
format_stamp: "Formatted at 2026-09-26 22:40:24 on dist-test-slave-j8sq"
server_key: "887ef6ce674b10aaf3dc45ae5f34b8a4"
server_key_iv: "6c22d1746507dff99fa16a0c55bb05bd"
server_key_version: "encryptionkey@0"
I20260926 22:40:24.219771 17232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:24.220503 17247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:24.220745 17232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:24.220849 17232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "943d8c6c3eb0490788f0a9587353e5b8"
format_stamp: "Formatted at 2026-09-26 22:40:24 on dist-test-slave-j8sq"
server_key: "887ef6ce674b10aaf3dc45ae5f34b8a4"
server_key_iv: "6c22d1746507dff99fa16a0c55bb05bd"
server_key_version: "encryptionkey@0"
I20260926 22:40:24.220920 17232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:24.241487 17232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:24.243441 17232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:24.243618 17232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:24.247469 17232 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:37725
I20260926 22:40:24.247520 17299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:37725 every 8 connection(s)
I20260926 22:40:24.247884 17232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:24.248701 17300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:24.250969 17300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8: Bootstrap starting.
I20260926 22:40:24.251536 17300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:24.251822 17300 log.cc:824] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:24.252478 17300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8: No bootstrap required, opened a new log
I20260926 22:40:24.254038 17300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943d8c6c3eb0490788f0a9587353e5b8" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37725 } }
I20260926 22:40:24.254195 17300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:24.254242 17300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 943d8c6c3eb0490788f0a9587353e5b8, State: Initialized, Role: FOLLOWER
I20260926 22:40:24.254395 17300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943d8c6c3eb0490788f0a9587353e5b8" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37725 } }
I20260926 22:40:24.254457 17300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:24.254495 17300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:24.254544 17300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:24.255019 17300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943d8c6c3eb0490788f0a9587353e5b8" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37725 } }
I20260926 22:40:24.255128 17300 leader_election.cc:302] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 943d8c6c3eb0490788f0a9587353e5b8; no voters:
I20260926 22:40:24.255322 17300 leader_election.cc:288] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:24.255396 17303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:24.255558 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 17232
I20260926 22:40:24.255580 17300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:24.255672 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:24.255772 17303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [term 1 LEADER]: Becoming Leader. State: Replica: 943d8c6c3eb0490788f0a9587353e5b8, State: Running, Role: LEADER
I20260926 22:40:24.255882 11695 external_mini_cluster.cc:1428] Setting key a254dce44d613a80d9f66f84751e928e
I20260926 22:40:24.255867 17303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943d8c6c3eb0490788f0a9587353e5b8" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37725 } }
I20260926 22:40:24.256183 17304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "943d8c6c3eb0490788f0a9587353e5b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943d8c6c3eb0490788f0a9587353e5b8" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37725 } } }
I20260926 22:40:24.256282 17304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:24.256186 17305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 943d8c6c3eb0490788f0a9587353e5b8. Latest consensus state: current_term: 1 leader_uuid: "943d8c6c3eb0490788f0a9587353e5b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943d8c6c3eb0490788f0a9587353e5b8" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37725 } } }
I20260926 22:40:24.256500 17305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:24.256623 17313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:24.257339 17313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:24.260123 17313 catalog_manager.cc:1347] Generated new cluster ID: b73d3cbb1bcc4506a58cab80c1acced6
I20260926 22:40:24.260185 17313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:24.266676 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:24.256487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.11.107.254:37725)
0926 22:40:24.256901 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:24.256912 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:24.256949 (+ 37us) server_negotiation.cc:407] Connection header received
0926 22:40:24.257069 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:24.257077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:24.257163 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:24.257300 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:24.258319 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.259678 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.260584 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.261003 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.261131 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:24.263488 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:24.263492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:24.263493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:24.263503 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:24.265103 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.265482 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.265484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.265484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.265570 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.265783 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.265784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.265784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.266085 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:24.266267 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:24.266348 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 22:40:24.266483 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":69,"threads_started":1}
I20260926 22:40:24.291234 17313 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:24.291988 17313 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:24.302924 17313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 943d8c6c3eb0490788f0a9587353e5b8: Generated new TSK 0
I20260926 22:40:24.303114 17313 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:40:24.330631 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:37725
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:24.411096 17326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:24.411283 17326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:24.411304 17326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:24.412366 17326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:24.412403 17326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:24.412428 17326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:24.413808 17326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:37725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:24.414045 17326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:24.414213 17326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:24.419509 17331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:24.419517 17332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:24.419682 17326 server_base.cc:1034] running on GCE node
W20260926 22:40:24.419524 17334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:24.419955 17326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:24.420229 17326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:24.421396 17326 hybrid_clock.cc:648] HybridClock initialized: now 1790462424421388 us; error 42 us; skew 500 ppm
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:24.424130 17326 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:40:24.424597 17326 webserver.cc:460] Webserver started at http://127.11.107.193:34113/ using document root <none> and password file <none>
I20260926 22:40:24.424831 17326 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:24.424873 17326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:24.424959 17326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:24.425979 17326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "086ab45e8b434ae0a6470fc5e0be0685"
format_stamp: "Formatted at 2026-09-26 22:40:24 on dist-test-slave-j8sq"
server_key: "05b5604622bf583d5b5c9ed4b86099ff"
server_key_iv: "362d79e16c89fbf425b2f273e7c4c38e"
server_key_version: "encryptionkey@0"
I20260926 22:40:24.426286 17326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "086ab45e8b434ae0a6470fc5e0be0685"
format_stamp: "Formatted at 2026-09-26 22:40:24 on dist-test-slave-j8sq"
server_key: "05b5604622bf583d5b5c9ed4b86099ff"
server_key_iv: "362d79e16c89fbf425b2f273e7c4c38e"
server_key_version: "encryptionkey@0"
I20260926 22:40:24.427453 17326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:24.428077 17341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:24.428359 17326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 22:40:24.428436 17326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "086ab45e8b434ae0a6470fc5e0be0685"
format_stamp: "Formatted at 2026-09-26 22:40:24 on dist-test-slave-j8sq"
server_key: "05b5604622bf583d5b5c9ed4b86099ff"
server_key_iv: "362d79e16c89fbf425b2f273e7c4c38e"
server_key_version: "encryptionkey@0"
I20260926 22:40:24.428488 17326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:24.444362 17326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:24.446761 17326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:24.446871 17326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:24.447045 17326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:24.447353 17326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:24.447386 17326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:24.447432 17326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:24.447458 17326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:24.453073 17326 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:37849
I20260926 22:40:24.453118 17454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:37849 every 8 connection(s)
I20260926 22:40:24.453423 17326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:24.455128 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 17326
I20260926 22:40:24.455226 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:24.455463 11695 external_mini_cluster.cc:1428] Setting key 2f9f4a6c089572177176b4fe924ab3d5
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
I20260926 22:40:24.463197 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:24.454111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:59723 (local address 127.11.107.254:37725)
0926 22:40:24.454172 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:24.454174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:24.454371 (+ 197us) server_negotiation.cc:407] Connection header received
0926 22:40:24.455390 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:24.455391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:24.455406 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:24.455467 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:24.456271 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.456865 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.458014 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.458229 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.458313 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:24.460420 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:24.460421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:24.460422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:24.460429 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:24.461818 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.462252 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.462253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.462253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.462314 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.462561 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.462562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.462563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.462682 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:24.462758 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:24.463022 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 22:40:24.463106 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 22:40:24.463388 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:24.454080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:37725 (local address 127.11.107.193:59723)
0926 22:40:24.454251 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:24.454294 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:24.455267 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:24.455536 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:24.455539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:24.455904 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:24.456178 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:24.456185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.456966 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.456967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:24.457913 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:24.457915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.458322 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.458324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:24.458461 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:24.458951 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:24.458964 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:24.460327 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:24.461936 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:24.461943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:24.461945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:24.462169 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:24.462379 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:24.462380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:24.462381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:24.462489 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:24.462749 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:24.462755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:24.462893 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 22:40:24.463002 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 22:40:24.463161 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260926 22:40:24.463907 17455 heartbeater.cc:347] Connected to a master server at 127.11.107.254:37725
I20260926 22:40:24.464015 17455 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:24.464210 17455 heartbeater.cc:511] Master 127.11.107.254:37725 requested a full tablet report, sending...
I20260926 22:40:24.464700 17264 ts_manager.cc:194] Registered new tserver with Master: 086ab45e8b434ae0a6470fc5e0be0685 (127.11.107.193:37849)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
I20260926 22:40:24.465323 17264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:59723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:40:24.479969 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:37725
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:24.559926 17460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:24.560129 17460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:24.560151 17460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:24.561331 17460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:24.561371 17460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:24.561421 17460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:24.562889 17460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:37725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:24.563136 17460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:24.563369 17460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:24.568148 17465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:24.568161 17468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:24.568157 17466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:24.568369 17460 server_base.cc:1034] running on GCE node
I20260926 22:40:24.568636 17460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:24.568915 17460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:24.570076 17460 hybrid_clock.cc:648] HybridClock initialized: now 1790462424570058 us; error 37 us; skew 500 ppm
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:24.572585 17460 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:40:24.573005 17460 webserver.cc:460] Webserver started at http://127.11.107.194:34769/ using document root <none> and password file <none>
I20260926 22:40:24.573238 17460 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:24.573305 17460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:24.573451 17460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:24.574527 17460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "994e160807c34adaa58bbea80014e10b"
format_stamp: "Formatted at 2026-09-26 22:40:24 on dist-test-slave-j8sq"
server_key: "293b0491d12e23c085aceb74e0f908b5"
server_key_iv: "d600c579992c6ed1257af9e4462bfeca"
server_key_version: "encryptionkey@0"
I20260926 22:40:24.574870 17460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "994e160807c34adaa58bbea80014e10b"
format_stamp: "Formatted at 2026-09-26 22:40:24 on dist-test-slave-j8sq"
server_key: "293b0491d12e23c085aceb74e0f908b5"
server_key_iv: "d600c579992c6ed1257af9e4462bfeca"
server_key_version: "encryptionkey@0"
I20260926 22:40:24.576115 17460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:40:24.576797 17475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:24.576953 17460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:24.577047 17460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "994e160807c34adaa58bbea80014e10b"
format_stamp: "Formatted at 2026-09-26 22:40:24 on dist-test-slave-j8sq"
server_key: "293b0491d12e23c085aceb74e0f908b5"
server_key_iv: "d600c579992c6ed1257af9e4462bfeca"
server_key_version: "encryptionkey@0"
I20260926 22:40:24.577121 17460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:24.584165 17460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:24.586280 17460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:24.586431 17460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:24.586661 17460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:24.586951 17460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:24.587003 17460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:24.587035 17460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:24.587061 17460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:24.592769 17460 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:34329
I20260926 22:40:24.592814 17588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:34329 every 8 connection(s)
I20260926 22:40:24.593122 17460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:40:24.594566 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 17460
I20260926 22:40:24.594696 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:24.594875 11695 external_mini_cluster.cc:1428] Setting key 03112ebbfb0409eaaf86c15ecad3229f
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
I20260926 22:40:24.603909 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:24.593805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:46237 (local address 127.11.107.254:37725)
0926 22:40:24.593901 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:24.593904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:24.594077 (+ 173us) server_negotiation.cc:407] Connection header received
0926 22:40:24.595160 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:24.595162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:24.595181 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:24.595239 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:24.595963 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.596731 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.598306 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.598490 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.598547 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:24.600914 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:24.600915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:24.600916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:24.600924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:24.602498 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.602914 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.602916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.602916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.602975 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.603227 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.603228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.603228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.603341 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:24.603417 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:24.603737 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 22:40:24.603820 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 22:40:24.604128 17591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:24.593765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:37725 (local address 127.11.107.194:46237)
0926 22:40:24.593947 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:24.593996 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:24.595003 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:24.595321 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:24.595326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:24.595685 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:24.595871 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:24.595876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.596818 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.596819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:24.598232 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:24.598235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.598570 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.598571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:24.598723 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:24.599224 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:24.599237 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:24.600783 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:24.602586 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:24.602592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:24.602595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:24.602823 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:24.603041 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:24.603041 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:24.603042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:24.603152 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:24.603408 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:24.603414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:24.603588 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 22:40:24.603704 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 22:40:24.603858 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
I20260926 22:40:24.604600 17589 heartbeater.cc:347] Connected to a master server at 127.11.107.254:37725
I20260926 22:40:24.604683 17589 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:24.604822 17589 heartbeater.cc:511] Master 127.11.107.254:37725 requested a full tablet report, sending...
I20260926 22:40:24.605123 17264 ts_manager.cc:194] Registered new tserver with Master: 994e160807c34adaa58bbea80014e10b (127.11.107.194:34329)
I20260926 22:40:24.605497 17264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:46237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:40:24.619964 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:37725
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:24.699837 17594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:24.700028 17594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:24.700050 17594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:24.701110 17594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:24.701149 17594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:24.701175 17594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:24.702539 17594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:37725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:24.702700 17594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:24.702857 17594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:24.707672 17602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:24.707682 17600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:24.707708 17594 server_base.cc:1034] running on GCE node
W20260926 22:40:24.707677 17599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:24.708074 17594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:24.708279 17594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:24.709450 17594 hybrid_clock.cc:648] HybridClock initialized: now 1790462424709437 us; error 25 us; skew 500 ppm
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:24.711908 17594 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:40:24.712353 17594 webserver.cc:460] Webserver started at http://127.11.107.195:36171/ using document root <none> and password file <none>
I20260926 22:40:24.712582 17594 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:24.712648 17594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:24.712801 17594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:24.713919 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "9d2dba4d6c144de792ca2b70b8f40a93"
format_stamp: "Formatted at 2026-09-26 22:40:24 on dist-test-slave-j8sq"
server_key: "aab9db04b2ee86272e554040f14f18f2"
server_key_iv: "261bef44de52980d499d7758d32586c7"
server_key_version: "encryptionkey@0"
I20260926 22:40:24.714295 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "9d2dba4d6c144de792ca2b70b8f40a93"
format_stamp: "Formatted at 2026-09-26 22:40:24 on dist-test-slave-j8sq"
server_key: "aab9db04b2ee86272e554040f14f18f2"
server_key_iv: "261bef44de52980d499d7758d32586c7"
server_key_version: "encryptionkey@0"
I20260926 22:40:24.715601 17594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:24.716292 17609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:24.716459 17594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:24.716559 17594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "9d2dba4d6c144de792ca2b70b8f40a93"
format_stamp: "Formatted at 2026-09-26 22:40:24 on dist-test-slave-j8sq"
server_key: "aab9db04b2ee86272e554040f14f18f2"
server_key_iv: "261bef44de52980d499d7758d32586c7"
server_key_version: "encryptionkey@0"
I20260926 22:40:24.716629 17594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:24.726589 17594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:24.728802 17594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:24.728951 17594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:24.729162 17594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:24.729453 17594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:24.729504 17594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:24.729545 17594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:24.729573 17594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:24.735204 17594 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:45717
I20260926 22:40:24.735241 17722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:45717 every 8 connection(s)
I20260926 22:40:24.735566 17594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:24.744030 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:24.736303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:59163 (local address 127.11.107.254:37725)
0926 22:40:24.736360 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:24.736362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:24.736607 (+ 245us) server_negotiation.cc:407] Connection header received
0926 22:40:24.737481 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:24.737482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:24.737490 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:24.737551 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:24.738179 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.738679 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.739594 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.739784 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.739837 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:24.741759 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:24.741760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:24.741760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:24.741767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:24.742960 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.743303 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.743304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.743304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.743352 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.743547 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.743547 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.743548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.743638 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:24.743696 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:24.743900 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 22:40:24.743957 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:40:24.744185 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:24.736297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:37725 (local address 127.11.107.195:59163)
0926 22:40:24.736479 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:24.736518 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:24.737400 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:24.737604 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:24.737607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:24.737929 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:24.738105 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:24.738109 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.738754 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.738754 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:24.739519 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:24.739521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.739855 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.739855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:24.739947 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:24.740434 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:24.740442 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:24.741681 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:24.743027 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:24.743032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:24.743033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:24.743241 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:24.743407 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:24.743408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:24.743408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:24.743495 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:24.743690 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:24.743694 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:24.743795 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 22:40:24.743873 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 22:40:24.743990 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":100,"threads_started":1}
I20260926 22:40:24.744261 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 17594
I20260926 22:40:24.744360 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:24.744530 11695 external_mini_cluster.cc:1428] Setting key 8093f12e98c4ac0d047f6a6adb6532d8
I20260926 22:40:24.744958 17723 heartbeater.cc:347] Connected to a master server at 127.11.107.254:37725
I20260926 22:40:24.745050 17723 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:24.745226 17723 heartbeater.cc:511] Master 127.11.107.254:37725 requested a full tablet report, sending...
I20260926 22:40:24.745613 17263 ts_manager.cc:194] Registered new tserver with Master: 9d2dba4d6c144de792ca2b70b8f40a93 (127.11.107.195:45717)
I20260926 22:40:24.746045 17263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:59163
I20260926 22:40:24.756340 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/test-admin.keytab.
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:24.771597 11695 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 22:40:24.773095 17730 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:24.772383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.11.107.193:37849)
0926 22:40:24.772543 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:24.772548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:24.772557 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:24.772581 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:24.772582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:24.772595 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:24.772660 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:24.773000 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37258: BlockingRecv error: recv got EOF from 127.0.0.1:37258 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
W20260926 22:40:24.773162 11695 init.cc:244] Error while opening credential cache '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/krb5cc)
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:24.774148 11695 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:24.774988 11695 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.193@KRBTEST.COM
I20260926 22:40:24.782385 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:24.775476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.11.107.193:37849)
0926 22:40:24.775528 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:24.775530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:24.775557 (+ 27us) server_negotiation.cc:407] Connection header received
0926 22:40:24.775615 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:24.775616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:24.775624 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:24.775672 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:24.776010 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.776710 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.777367 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.777660 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.777713 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:24.779650 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:24.779659 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:24.779659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:24.779667 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:24.781290 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.781615 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.781617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.781618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.781676 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.781872 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.781873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.781874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.782002 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:24.782095 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:24.782184 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 22:40:24.782240 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 22:40:24.784014 17730 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:24.783518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.11.107.193:37849)
0926 22:40:24.783583 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:24.783585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:24.783629 (+ 44us) server_negotiation.cc:407] Connection header received
0926 22:40:24.783672 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:24.783672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:24.783680 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:24.783722 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:24.783954 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37278: BlockingRecv error: recv got EOF from 127.0.0.1:37278 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 22:40:24.784078 11695 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/krb5cc' of type FILE
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:24.784943 11695 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:24.785725 11695 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.193@KRBTEST.COM
I20260926 22:40:24.791805 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:24.786218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.11.107.193:37849)
0926 22:40:24.786271 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:24.786272 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:24.786299 (+ 27us) server_negotiation.cc:407] Connection header received
0926 22:40:24.786352 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:24.786353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:24.786361 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:24.786403 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:24.786696 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.787129 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.787740 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.787914 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.787960 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:24.789737 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:24.789738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:24.789738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:24.789744 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:24.790880 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.791162 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.791163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.791163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.791206 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.791345 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.791345 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.791346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.791420 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:24.791496 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:24.791605 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 22:40:24.791669 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 22:40:24.793292 17730 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:24.792829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.11.107.193:37849)
0926 22:40:24.792885 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:24.792886 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:24.792917 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:40:24.792964 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:24.792964 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:24.792971 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:24.793013 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:24.793253 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37288: BlockingRecv error: recv got EOF from 127.0.0.1:37288 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 22:40:24.793370 11695 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462400949426-11695-0/krb5kdc/krb5cc' of type FILE
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:24.794467 11695 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:24.795344 11695 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.193@KRBTEST.COM
I20260926 22:40:24.801985 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:24.795835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.11.107.193:37849)
0926 22:40:24.795890 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:24.795892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:24.795971 (+ 79us) server_negotiation.cc:407] Connection header received
0926 22:40:24.796042 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:24.796043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:24.796052 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:24.796121 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:24.796479 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.796940 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.797622 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:24.797813 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:24.797943 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:24.799775 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:24.799776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:24.799776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:24.799783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:24.800897 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.801231 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.801232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.801232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.801286 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:24.801479 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:24.801480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:24.801481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:24.801572 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:24.801673 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:24.801800 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 22:40:24.801892 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:40:24.802525 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 17326
I20260926 22:40:24.808291 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 17460
I20260926 22:40:24.812134 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 17594
I20260926 22:40:24.816231 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 17232
2026-09-26T22:40:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1844 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17743](info): set up 2 sockets
Sep 26 22:40:24 dist-test-slave-j8sq krb5kdc[17743](info): commencing operation
krb5kdc: starting...
W20260926 22:40:25.843199 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:25 dist-test-slave-j8sq krb5kdc[17743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.254.keytab.
Entry for principal oryx/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.254.keytab.
I20260926 22:40:25.956674 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:32867
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.254.keytab
--principal=oryx/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:32867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:26.036296 17756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:26.036531 17756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:26.036571 17756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:26.037760 17756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:26.037823 17756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:26.037850 17756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:26.037896 17756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:26.037925 17756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:26.039387 17756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:32867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.254.keytab
--principal=oryx/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:32867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:26.039674 17756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:26.039902 17756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:26.044775 17762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:26.044770 17761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:26.044767 17764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:26.044945 17756 server_base.cc:1034] running on GCE node
I20260926 22:40:26.045252 17756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:26.045578 17756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:26.046715 17756 hybrid_clock.cc:648] HybridClock initialized: now 1790462426046712 us; error 36 us; skew 500 ppm
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462426, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:26.049276 17756 init.cc:377] Logged in from keytab as oryx/127.11.107.254@KRBTEST.COM (short username oryx)
I20260926 22:40:26.049649 17756 webserver.cc:460] Webserver started at http://127.11.107.254:42357/ using document root <none> and password file <none>
I20260926 22:40:26.049885 17756 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:26.049944 17756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:26.050032 17756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:26.051038 17756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "ce9c66c085544744a0b3ba6679cd761e"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "0c27afd6b47fe5e8ef3f2061e6607e30"
server_key_iv: "3f0196b12d6206c3e7c1963a35033769"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.051333 17756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "ce9c66c085544744a0b3ba6679cd761e"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "0c27afd6b47fe5e8ef3f2061e6607e30"
server_key_iv: "3f0196b12d6206c3e7c1963a35033769"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.052539 17756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:26.053229 17771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.053408 17756 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:26.053506 17756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "ce9c66c085544744a0b3ba6679cd761e"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "0c27afd6b47fe5e8ef3f2061e6607e30"
server_key_iv: "3f0196b12d6206c3e7c1963a35033769"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.053584 17756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:26.066476 17756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:26.068728 17756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:26.068881 17756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:26.072834 17756 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:32867
I20260926 22:40:26.072866 17823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:32867 every 8 connection(s)
I20260926 22:40:26.073225 17756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:26.074337 17824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:26.076598 17824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e: Bootstrap starting.
I20260926 22:40:26.077178 17824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:26.077467 17824 log.cc:824] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:26.078225 17824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e: No bootstrap required, opened a new log
I20260926 22:40:26.079767 17824 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } }
I20260926 22:40:26.079900 17824 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:26.079945 17824 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9c66c085544744a0b3ba6679cd761e, State: Initialized, Role: FOLLOWER
I20260926 22:40:26.080094 17824 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } }
I20260926 22:40:26.080175 17824 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:26.080228 17824 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:26.080276 17824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:26.081243 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 17756
I20260926 22:40:26.081238 17824 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } }
I20260926 22:40:26.081352 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:26.081346 17824 leader_election.cc:302] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce9c66c085544744a0b3ba6679cd761e; no voters:
I20260926 22:40:26.081504 17824 leader_election.cc:288] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:26.081555 17827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:26.081581 11695 external_mini_cluster.cc:1428] Setting key 260d85fc9e55cfc2c5150a4bcc4a541a
I20260926 22:40:26.081712 17827 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [term 1 LEADER]: Becoming Leader. State: Replica: ce9c66c085544744a0b3ba6679cd761e, State: Running, Role: LEADER
I20260926 22:40:26.081765 17824 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:26.081804 17827 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } }
I20260926 22:40:26.082113 17827 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce9c66c085544744a0b3ba6679cd761e. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } }
I20260926 22:40:26.082194 17827 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:26.082118 17829 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } }
I20260926 22:40:26.082353 17829 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:26.082405 17833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:26.082934 17833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:26.084676 17833 catalog_manager.cc:1347] Generated new cluster ID: c4f1ae9664b646ac9c393356187e9503
I20260926 22:40:26.084730 17833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
I20260926 22:40:26.093492 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.082229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.11.107.254:32867)
0926 22:40:26.082401 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:26.082413 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:26.082631 (+ 218us) server_negotiation.cc:407] Connection header received
0926 22:40:26.082761 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:26.082768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:26.082823 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:26.083786 (+ 963us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:26.083891 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.085344 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.086436 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.087063 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.087191 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.089907 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:26.089911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:26.089912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:26.089924 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:26.091864 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.092282 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.092284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.092285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.092402 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.092615 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.092615 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.092616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.092893 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:26.093035 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:26.093164 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 22:40:26.093287 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260926 22:40:26.106320 17833 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:26.106851 17833 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:26.113089 17833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e: Generated new TSK 0
I20260926 22:40:26.113299 17833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab.
Entry for principal oryx/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab.
I20260926 22:40:26.146449 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:32867
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab
--principal=oryx/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:26.253242 17850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:26.253543 17850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:26.253580 17850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:26.255362 17850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:26.255414 17850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:26.255451 17850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:26.257654 17850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab
--principal=oryx/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:32867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:26.257949 17850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:26.258186 17850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:26.263144 17858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:26.263185 17856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:26.263309 17850 server_base.cc:1034] running on GCE node
W20260926 22:40:26.263401 17855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:26.263629 17850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:26.263911 17850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:26.265072 17850 hybrid_clock.cc:648] HybridClock initialized: now 1790462426265054 us; error 34 us; skew 500 ppm
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462426, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:26.267418 17850 init.cc:377] Logged in from keytab as oryx/127.11.107.193@KRBTEST.COM (short username oryx)
I20260926 22:40:26.267827 17850 webserver.cc:460] Webserver started at http://127.11.107.193:39411/ using document root <none> and password file <none>
I20260926 22:40:26.268051 17850 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:26.268095 17850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:26.268268 17850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:26.269265 17850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "5c80eb7981614c99afa3149d5f4cc225"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "43e1ade7a4d12bc582243096f40e424a"
server_key_iv: "48005147c4c028ad4fe551a09b0e23b8"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.269610 17850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "5c80eb7981614c99afa3149d5f4cc225"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "43e1ade7a4d12bc582243096f40e424a"
server_key_iv: "48005147c4c028ad4fe551a09b0e23b8"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.270949 17850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:26.271700 17865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.271893 17850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:26.271981 17850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "5c80eb7981614c99afa3149d5f4cc225"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "43e1ade7a4d12bc582243096f40e424a"
server_key_iv: "48005147c4c028ad4fe551a09b0e23b8"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.272053 17850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:26.282601 17850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:26.284507 17850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:26.284677 17850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:26.284925 17850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:26.285181 17850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:26.285213 17850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.285236 17850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:26.285249 17850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.290863 17850 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:44109
I20260926 22:40:26.290953 17978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:44109 every 8 connection(s)
I20260926 22:40:26.291250 17850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:26.291581 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 17850
I20260926 22:40:26.291706 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:26.291885 11695 external_mini_cluster.cc:1428] Setting key 69cb87cd8efb01efa80e1abcde246860
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462426, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.11.107.194@KRBTEST.COM; defaulting to no policy
I20260926 22:40:26.301537 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.292190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:38949 (local address 127.11.107.254:32867)
0926 22:40:26.292261 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:26.292263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:26.292272 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:26.292727 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:26.292728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:26.292743 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:26.292813 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:26.293820 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.294358 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.295890 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.296134 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.296192 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.298714 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:26.298715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:26.298715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:26.298723 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:26.300102 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.300541 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.300542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.300543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.300601 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.300851 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.300852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.300853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.300964 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:26.301169 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:26.301357 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 22:40:26.301450 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 22:40:26.301707 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.291948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.193:38949)
0926 22:40:26.292106 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:26.292147 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:26.292632 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:26.292867 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:26.292870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:26.293445 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:26.293714 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.293722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.294454 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.294456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.295802 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.295806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.296229 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.296230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.296382 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.296859 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:26.296870 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:26.298612 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:26.300206 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.300213 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.300215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.300451 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.300679 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.300680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.300680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.300783 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.301032 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:26.301039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:26.301198 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 22:40:26.301324 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 22:40:26.301474 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260926 22:40:26.302242 17979 heartbeater.cc:347] Connected to a master server at 127.11.107.254:32867
I20260926 22:40:26.302320 17979 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:26.302544 17979 heartbeater.cc:511] Master 127.11.107.254:32867 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.107.194@KRBTEST.COM" created.
I20260926 22:40:26.303059 17788 ts_manager.cc:194] Registered new tserver with Master: 5c80eb7981614c99afa3149d5f4cc225 (127.11.107.193:44109)
I20260926 22:40:26.303771 17788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:38949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab.
Entry for principal oryx/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab.
I20260926 22:40:26.316941 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:32867
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab
--principal=oryx/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:26.397011 17984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:26.397251 17984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:26.397289 17984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:26.398449 17984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:26.398514 17984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:26.398558 17984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:26.399991 17984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab
--principal=oryx/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:32867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:26.400252 17984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:26.400484 17984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:26.405354 17992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:26.405411 17989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:26.405475 17990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:26.405774 17984 server_base.cc:1034] running on GCE node
I20260926 22:40:26.406000 17984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:26.406234 17984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:26.407368 17984 hybrid_clock.cc:648] HybridClock initialized: now 1790462426407355 us; error 28 us; skew 500 ppm
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462426, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:26.409937 17984 init.cc:377] Logged in from keytab as oryx/127.11.107.194@KRBTEST.COM (short username oryx)
I20260926 22:40:26.410392 17984 webserver.cc:460] Webserver started at http://127.11.107.194:40725/ using document root <none> and password file <none>
I20260926 22:40:26.410627 17984 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:26.410677 17984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:26.410801 17984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:26.411827 17984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "fcf3b707309165b2be0384c89cd70592"
server_key_iv: "8cb3006888262c8127a8a8bd2188744d"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.412171 17984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "fcf3b707309165b2be0384c89cd70592"
server_key_iv: "8cb3006888262c8127a8a8bd2188744d"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.413496 17984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:26.414279 17999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.414456 17984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.414525 17984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "fcf3b707309165b2be0384c89cd70592"
server_key_iv: "8cb3006888262c8127a8a8bd2188744d"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.414614 17984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:26.435171 17984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:26.437552 17984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:26.437732 17984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:26.438000 17984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:26.438323 17984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:26.438354 17984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.438405 17984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:26.438431 17984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.444244 17984 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:35919
I20260926 22:40:26.444303 18112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:35919 every 8 connection(s)
I20260926 22:40:26.444650 17984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462426, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
I20260926 22:40:26.451820 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 17984
I20260926 22:40:26.451923 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:26.452117 11695 external_mini_cluster.cc:1428] Setting key d6d99d2d1abb4f989429aee2b6fd2fb8
I20260926 22:40:26.453399 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.445397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:39765 (local address 127.11.107.254:32867)
0926 22:40:26.445486 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:26.445488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:26.445638 (+ 150us) server_negotiation.cc:407] Connection header received
0926 22:40:26.446104 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:26.446105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:26.446117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:26.446171 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:26.447018 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.447500 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.448418 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.448596 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.448667 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.450714 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:26.450715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:26.450715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:26.450721 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:26.451882 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.452443 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.452444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.452445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.452506 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.452776 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.452777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.452777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.452901 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:26.452952 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:26.453275 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 22:40:26.453340 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 22:40:26.453686 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.445355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.194:39765)
0926 22:40:26.445509 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:26.445544 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:26.446020 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:26.446223 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:26.446226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:26.446760 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:26.446943 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.446948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.447573 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.447574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.448348 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.448350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.448694 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.448695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.448795 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.449292 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:26.449300 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:26.450621 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:26.452089 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.452096 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.452098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.452389 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.452575 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.452575 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.452576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.452698 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.452949 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:26.452955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:26.453116 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 22:40:26.453249 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 22:40:26.453421 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":97,"threads_started":1}
I20260926 22:40:26.454154 18113 heartbeater.cc:347] Connected to a master server at 127.11.107.254:32867
I20260926 22:40:26.454241 18113 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:26.454427 18113 heartbeater.cc:511] Master 127.11.107.254:32867 requested a full tablet report, sending...
I20260926 22:40:26.454804 17788 ts_manager.cc:194] Registered new tserver with Master: b0ce7f00f3424cf0982c2b4de5e5cdeb (127.11.107.194:35919)
I20260926 22:40:26.455180 17788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:39765
WARNING: no policy specified for oryx/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab.
Entry for principal oryx/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab.
I20260926 22:40:26.476114 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:32867
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab
--principal=oryx/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:26.555534 18118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:26.555809 18118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:26.555856 18118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:26.557111 18118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:26.557171 18118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:26.557219 18118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:26.558802 18118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab
--principal=oryx/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:32867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:26.559069 18118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:26.559312 18118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:26.564155 18126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:26.564193 18123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:26.564157 18124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:26.564364 18118 server_base.cc:1034] running on GCE node
I20260926 22:40:26.564548 18118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:26.564735 18118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:26.565897 18118 hybrid_clock.cc:648] HybridClock initialized: now 1790462426565887 us; error 27 us; skew 500 ppm
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462426, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:26.568253 18118 init.cc:377] Logged in from keytab as oryx/127.11.107.195@KRBTEST.COM (short username oryx)
I20260926 22:40:26.568614 18118 webserver.cc:460] Webserver started at http://127.11.107.195:43501/ using document root <none> and password file <none>
I20260926 22:40:26.568801 18118 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:26.568842 18118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:26.568925 18118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:26.569944 18118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "7ea6064487bc48d7a050babd80852d04"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "61c6cc8ec44d9c90531a21c2941d99ac"
server_key_iv: "58b71ac71b06c54cf14fad025fa07b39"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.570250 18118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "7ea6064487bc48d7a050babd80852d04"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "61c6cc8ec44d9c90531a21c2941d99ac"
server_key_iv: "58b71ac71b06c54cf14fad025fa07b39"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.571508 18118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:26.572122 18133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.572284 18118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:26.572365 18118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "7ea6064487bc48d7a050babd80852d04"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "61c6cc8ec44d9c90531a21c2941d99ac"
server_key_iv: "58b71ac71b06c54cf14fad025fa07b39"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.572419 18118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:26.592770 18118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:26.595108 18118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:26.595232 18118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:26.595415 18118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:26.595739 18118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:26.595777 18118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.595820 18118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:26.595847 18118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.601722 18118 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:33873
I20260926 22:40:26.601783 18246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:33873 every 8 connection(s)
I20260926 22:40:26.602131 18118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462426, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
I20260926 22:40:26.610582 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.602859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:56373 (local address 127.11.107.254:32867)
0926 22:40:26.602923 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:26.602925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:26.603104 (+ 179us) server_negotiation.cc:407] Connection header received
0926 22:40:26.603577 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:26.603578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:26.603587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:26.603638 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:26.604503 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.604992 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.605970 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.606145 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.606198 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.608074 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:26.608075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:26.608075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:26.608081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:26.609426 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.609759 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.609760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.609761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.609809 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.610075 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.610076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.610076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.610178 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:26.610230 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:26.610463 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 22:40:26.610522 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 22:40:26.610793 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.602831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.195:56373)
0926 22:40:26.602984 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:26.603024 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:26.603499 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:26.603703 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:26.603706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:26.604247 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:26.604430 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.604435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.605067 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.605068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.605896 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.605898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.606213 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.606214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.606314 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.606716 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:26.606724 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:26.607998 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:26.609491 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.609495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.609497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.609680 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.609922 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.609923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.609923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.610021 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.610237 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:26.610241 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:26.610350 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 22:40:26.610435 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 22:40:26.610544 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260926 22:40:26.611061 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 18118
I20260926 22:40:26.611177 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:26.611378 11695 external_mini_cluster.cc:1428] Setting key 4bece6a4ee67b6ba79300be8be37b386
I20260926 22:40:26.611371 18247 heartbeater.cc:347] Connected to a master server at 127.11.107.254:32867
I20260926 22:40:26.611459 18247 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:26.611631 18247 heartbeater.cc:511] Master 127.11.107.254:32867 requested a full tablet report, sending...
I20260926 22:40:26.611943 17788 ts_manager.cc:194] Registered new tserver with Master: 7ea6064487bc48d7a050babd80852d04 (127.11.107.195:33873)
I20260926 22:40:26.612334 17788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:56373
I20260926 22:40:26.612450 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.107.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.107.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.107.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.253.keytab.
Entry for principal oryx/127.11.107.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.253.keytab.
I20260926 22:40:26.634801 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.253:36977
--webserver_interface=127.11.107.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.253.keytab
--principal=oryx/127.11.107.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:32867,127.11.107.253:36977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:26.714851 18252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:26.715121 18252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:26.715159 18252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:26.716360 18252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:26.716434 18252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:26.716459 18252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:26.716488 18252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:26.716517 18252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:26.718077 18252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:32867,127.11.107.253:36977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.253.keytab
--principal=oryx/127.11.107.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.253:36977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.107.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
W20260926 22:40:26.718221 18252 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.107.254:32867,127.11.107.253:36977'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 22:40:26.718377 18252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:26.718600 18252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:26.723248 18257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:26.723255 18260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:26.723459 18252 server_base.cc:1034] running on GCE node
W20260926 22:40:26.723275 18258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:26.723775 18252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:26.724094 18252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:26.725255 18252 hybrid_clock.cc:648] HybridClock initialized: now 1790462426725265 us; error 48 us; skew 500 ppm
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462426, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:26.727735 18252 init.cc:377] Logged in from keytab as oryx/127.11.107.253@KRBTEST.COM (short username oryx)
I20260926 22:40:26.728139 18252 webserver.cc:460] Webserver started at http://127.11.107.253:45797/ using document root <none> and password file <none>
I20260926 22:40:26.728365 18252 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:26.728431 18252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:26.728578 18252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:26.729647 18252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/data/instance:
uuid: "ab6c05c274ba4e3989741959ee16ed82"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "852d4c39c120144bf4dc81a17568064b"
server_key_iv: "2ee37e25ad158d88f30cce6232ffd7b8"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.730044 18252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/wal/instance:
uuid: "ab6c05c274ba4e3989741959ee16ed82"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "852d4c39c120144bf4dc81a17568064b"
server_key_iv: "2ee37e25ad158d88f30cce6232ffd7b8"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.731346 18252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:26.732051 18267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.732215 18252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:26.732326 18252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/wal
uuid: "ab6c05c274ba4e3989741959ee16ed82"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "852d4c39c120144bf4dc81a17568064b"
server_key_iv: "2ee37e25ad158d88f30cce6232ffd7b8"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.732406 18252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:26.738727 18252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:26.740885 18252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:26.741034 18252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:26.744874 18252 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.253:36977
I20260926 22:40:26.744899 18319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.253:36977 every 8 connection(s)
I20260926 22:40:26.745229 18252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/data/info.pb
I20260926 22:40:26.746236 18320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:26.747653 18320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:40:26.749133 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 18252
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462426, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.253@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.107.253@KRBTEST.COM
I20260926 22:40:26.757238 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.748309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.11.107.254:32867)
0926 22:40:26.748386 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:26.748389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:26.748601 (+ 212us) server_negotiation.cc:407] Connection header received
0926 22:40:26.749063 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:26.749065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:26.749078 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:26.749151 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:26.749915 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.750517 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.751484 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.751676 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.751728 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.754161 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:26.754162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:26.754163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:26.754174 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:26.755594 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.756089 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.756090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.756091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.756154 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.756450 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.756451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.756452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.756597 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:26.756665 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:26.757094 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 22:40:26.757171 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 22:40:26.757437 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.748300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:37710)
0926 22:40:26.748485 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:26.748518 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:26.748969 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:26.749248 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:26.749258 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:26.749480 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:26.749779 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.749785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.750628 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.750629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.751369 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.751371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.751753 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.751755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.751888 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.752461 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:26.752470 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:26.754067 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:26.755695 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.755701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.755704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.756014 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.756223 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.756224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.756224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.756362 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.756667 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:26.756673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:26.756861 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 22:40:26.757006 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 22:40:26.757180 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":112,"threads_started":1}
I20260926 22:40:26.758104 18320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:40:26.758303 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.749646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.11.107.253:36977)
0926 22:40:26.749794 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:26.749798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:26.749817 (+ 19us) server_negotiation.cc:407] Connection header received
0926 22:40:26.749917 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:26.749919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:26.749940 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:26.750035 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:26.750612 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.751346 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.752276 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.752645 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.752709 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.755043 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:26.755052 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:26.755052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:26.755060 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:26.756879 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.757273 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.757276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.757276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.757330 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.757613 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.757614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.757614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.757768 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:26.757947 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:26.758055 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 22:40:26.758138 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462426, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.253@KRBTEST.COM for oryx/127.11.107.253@KRBTEST.COM
I20260926 22:40:26.764065 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.758391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:36977 (local address 127.0.0.1:36238)
0926 22:40:26.758425 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:26.758436 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:26.758476 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:26.758638 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:26.758639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:26.758745 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:26.758789 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.758790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.759394 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.759395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.759839 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.759840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.760177 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.760178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.760225 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.760537 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:26.760541 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:26.761574 (+ 1033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:26.763145 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.763146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.763146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.763303 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.763493 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.763494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.763494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.763543 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.763780 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:26.763781 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:26.763814 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 22:40:26.763872 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 22:40:26.763959 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 22:40:26.764111 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.758412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.11.107.253:36977)
0926 22:40:26.758525 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:26.758527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:26.758531 (+ 4us) server_negotiation.cc:407] Connection header received
0926 22:40:26.758544 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:26.758545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:26.758570 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:26.758610 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:26.758895 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.759332 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.759915 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.760101 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.760156 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.761702 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:26.761703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:26.761704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:26.761710 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:26.763049 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.763382 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.763383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.763384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.763432 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.763612 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.763613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.763613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.763717 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:26.763788 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:26.763899 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 22:40:26.764004 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":104}
I20260926 22:40:26.765293 18320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: Bootstrap starting.
I20260926 22:40:26.766103 18320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:26.766326 18320 log.cc:824] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:26.766956 18320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: No bootstrap required, opened a new log
I20260926 22:40:26.768246 18320 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } }
I20260926 22:40:26.768358 18320 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:26.768379 18320 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6c05c274ba4e3989741959ee16ed82, State: Initialized, Role: FOLLOWER
I20260926 22:40:26.768493 18320 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } }
I20260926 22:40:26.768764 18320 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:26.768754 18326 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } } }
I20260926 22:40:26.768898 18326 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 22:40:26.770663 18337 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:40:26.771193 18337 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:40:26.773092 18252 master_runner.cc:428] Detected that this master ab6c05c274ba4e3989741959ee16ed82 is joining an existing cluster
I20260926 22:40:26.773154 18252 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.107.253:36977
I20260926 22:40:26.780493 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.774371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.11.107.254:32867)
0926 22:40:26.774428 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:26.774430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:26.774438 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:40:26.774554 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:26.774557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:26.774571 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:26.774611 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:26.774972 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.775470 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.776519 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.776698 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.776749 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.777684 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:26.777685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:26.777686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:26.777691 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:26.779293 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.779656 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.779657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.779658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.779717 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.779943 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.779944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.779944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.780057 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:26.780133 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:26.780301 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 22:40:26.780382 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:40:26.780829 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.774178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:37722)
0926 22:40:26.774352 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:26.774362 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:26.774482 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:26.774615 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:26.774616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:26.774782 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:26.774865 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.774867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.775544 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.775545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.776445 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.776448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.776779 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.776780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.776848 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.777227 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:26.777232 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:26.777595 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:26.779373 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.779374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.779374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.779578 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.779789 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.779790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.779790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.779860 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.780125 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:26.780126 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:26.780174 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 22:40:26.780622 (+ 448us) client_negotiation.cc:240] Negotiation successful
0926 22:40:26.780721 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260926 22:40:26.784162 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.774196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.11.107.253:36977)
0926 22:40:26.774260 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:26.774262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:26.778028 (+ 3766us) server_negotiation.cc:407] Connection header received
0926 22:40:26.778074 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:26.778075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:26.778082 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:26.778109 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:26.778481 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.778956 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.779853 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.780134 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.780210 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.781470 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:26.781471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:26.781471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:26.781477 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:26.783037 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.783361 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.783362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.783362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.783417 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.783655 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.783656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.783656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.783762 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:26.783835 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:26.783963 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 22:40:26.784066 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":39}
I20260926 22:40:26.784339 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.774246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:36977 (local address 127.0.0.1:36242)
0926 22:40:26.777953 (+ 3707us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:26.777965 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:26.778021 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:26.778168 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:26.778169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:26.778356 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:26.778406 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.778407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.779056 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.779057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.779792 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.779794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.780217 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.780218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.780294 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.780827 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:26.780832 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:26.781388 (+ 556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:26.783114 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.783115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.783116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.783299 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.783476 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.783477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.783477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.783584 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.783827 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:26.783827 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:26.783862 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 22:40:26.783929 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 22:40:26.784264 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3692,"thread_start_us":64,"threads_started":1}
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462426, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.254@KRBTEST.COM for oryx/127.11.107.253@KRBTEST.COM
I20260926 22:40:26.791576 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.784023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.11.107.253:36977)
0926 22:40:26.784134 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:26.784136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:26.784155 (+ 19us) server_negotiation.cc:407] Connection header received
0926 22:40:26.784420 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:26.784421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:26.784431 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:26.784493 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:26.785006 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.785481 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.786648 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.786835 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.786907 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.788845 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:26.788846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:26.788847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:26.788853 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:26.790348 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.790746 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.790747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.790747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.790807 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.791007 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.791008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.791008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.791116 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:26.791191 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:26.791384 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 22:40:26.791465 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260926 22:40:26.791576 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.783842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:36977 (local address 127.0.0.1:36258)
0926 22:40:26.784035 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:26.784082 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:26.784141 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:26.784540 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:26.784545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:26.784876 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:26.784929 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.784931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.785610 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.785611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.786576 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.786579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.786910 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.786911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.787000 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.787380 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:26.787384 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:26.788758 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:26.790447 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.790456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.790458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.790667 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.790874 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.790875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.790876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.790936 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.791185 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:26.791194 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:26.791246 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 22:40:26.791346 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 22:40:26.791468 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
I20260926 22:40:26.792502 17788 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.107.253:36977: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82"
member_type: NON_VOTER
last_known_addr {
host: "127.11.107.253"
port: 36977
}
attrs {
promote: true
}
}
dest_uuid: "ce9c66c085544744a0b3ba6679cd761e"
cas_config_opid_index: -1
I20260926 22:40:26.792831 17788 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true } }
I20260926 22:40:26.793354 18347 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ab6c05c274ba4e3989741959ee16ed82 (127.11.107.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true } } }
I20260926 22:40:26.793689 18294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:26.794459 18347 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.107.253:36977
I20260926 22:40:26.794497 18294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 FOLLOWER]: Refusing update from remote peer ce9c66c085544744a0b3ba6679cd761e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 22:40:26.794703 18252 master.cc:537] Master@127.11.107.253:36977 shutting down...
I20260926 22:40:26.794708 18347 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true } } }
I20260926 22:40:26.794777 18347 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:26.794878 18347 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 22:40:26.795580 17775 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 22:40:26.796007 17775 consensus_peers.cc:487] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e -> Peer ab6c05c274ba4e3989741959ee16ed82 (127.11.107.253:36977): Couldn't send request to peer ab6c05c274ba4e3989741959ee16ed82. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 22:40:26.797274 18252 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:26.797359 18252 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:26.797410 18252 tablet_replica.cc:330] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: stopping tablet replica
I20260926 22:40:26.810142 18252 master.cc:552] Master@127.11.107.253:36977 shutdown complete.
I20260926 22:40:26.810210 18252 master_runner.cc:305] Clearing existing system tablet
I20260926 22:40:26.810336 18252 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:26.810393 18252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:26.810998 18252 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:26.811491 18352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.811771 18252 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:26.811836 18252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 22:40:26.811864 18252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/wal
uuid: "ab6c05c274ba4e3989741959ee16ed82"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "852d4c39c120144bf4dc81a17568064b"
server_key_iv: "2ee37e25ad158d88f30cce6232ffd7b8"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.812878 18252 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:26.814220 18252 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 22:40:26.814296 18252 log.cc:1196] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 22:40:26.814512 18252 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: Deleting consensus metadata
I20260926 22:40:26.814641 18252 master_runner.cc:315] Copying system tablet from 127.11.107.254:32867
I20260926 22:40:26.815249 18252 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: tablet copy: Beginning tablet copy session from remote peer at address 127.11.107.254:32867
I20260926 22:40:26.821719 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.815521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:37730)
0926 22:40:26.815675 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:26.815686 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:26.815755 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:26.815889 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:26.815890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:26.816082 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:26.816155 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.816158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.816841 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.816843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.817734 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:26.817736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.818157 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.818158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:26.818228 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.818648 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:26.818653 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:26.819050 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:26.820806 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.820807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.820807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.820970 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.821180 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:26.821180 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:26.821180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:26.821231 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:26.821486 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:26.821487 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:26.821521 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 22:40:26.821570 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:40:26.821644 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260926 22:40:26.821782 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.815616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.11.107.254:32867)
0926 22:40:26.815667 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:26.815669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:26.815746 (+ 77us) server_negotiation.cc:407] Connection header received
0926 22:40:26.815815 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:26.815815 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:26.815823 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:26.815867 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:26.816223 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.816771 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.817808 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.818077 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.818133 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.819144 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:26.819145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:26.819146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:26.819152 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:26.820723 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.821040 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.821041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.821042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.821103 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.821291 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.821292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.821292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.821425 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:26.821492 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:26.821616 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 22:40:26.821691 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:40:26.822206 17808 tablet_copy_service.cc:140] P ce9c66c085544744a0b3ba6679cd761e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ab6c05c274ba4e3989741959ee16ed82 ({username='oryx', principal='oryx/127.11.107.253@KRBTEST.COM'} at 127.0.0.1:37730)
I20260926 22:40:26.822300 17808 tablet_copy_service.cc:161] P ce9c66c085544744a0b3ba6679cd761e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ab6c05c274ba4e3989741959ee16ed82 at {username='oryx', principal='oryx/127.11.107.253@KRBTEST.COM'} at 127.0.0.1:37730: session id = ab6c05c274ba4e3989741959ee16ed82-00000000000000000000000000000000
I20260926 22:40:26.823068 17808 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 22:40:26.823673 18252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 22:40:26.824599 18252 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: tablet copy: Starting download of 0 data blocks...
I20260926 22:40:26.824761 18252 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: tablet copy: Starting download of 1 WAL segments...
I20260926 22:40:26.825625 18252 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 22:40:26.826535 17808 tablet_copy_service.cc:342] P ce9c66c085544744a0b3ba6679cd761e: Request end of tablet copy session ab6c05c274ba4e3989741959ee16ed82-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.107.253@KRBTEST.COM'} at 127.0.0.1:37730
I20260926 22:40:26.826591 17808 tablet_copy_service.cc:433] P ce9c66c085544744a0b3ba6679cd761e: ending tablet copy session ab6c05c274ba4e3989741959ee16ed82-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ab6c05c274ba4e3989741959ee16ed82
I20260926 22:40:26.829304 18252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:26.830430 18359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:26.830494 18252 server_base.cc:1034] running on GCE node
W20260926 22:40:26.830472 18362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:26.830615 18360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:26.830770 18252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:26.830935 18252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:26.832062 18252 hybrid_clock.cc:648] HybridClock initialized: now 1790462426832053 us; error 25 us; skew 500 ppm
Sep 26 22:40:26 dist-test-slave-j8sq krb5kdc[17743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462426, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:26.833355 18252 init.cc:377] Logged in from keytab as oryx/127.11.107.253@KRBTEST.COM (short username oryx)
I20260926 22:40:26.833618 18252 webserver.cc:460] Webserver started at http://127.11.107.253:42371/ using document root <none> and password file <none>
I20260926 22:40:26.833755 18252 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:26.833786 18252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:26.834441 18252 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.835008 18369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:26.835127 18252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:26.835179 18252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/wal
uuid: "ab6c05c274ba4e3989741959ee16ed82"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "852d4c39c120144bf4dc81a17568064b"
server_key_iv: "2ee37e25ad158d88f30cce6232ffd7b8"
server_key_version: "encryptionkey@0"
I20260926 22:40:26.835237 18252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:26.858119 18252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:26.859979 18252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:26.863060 18252 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.253:36977
I20260926 22:40:26.863076 18421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.253:36977 every 8 connection(s)
I20260926 22:40:26.863402 18252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-1/data/info.pb
I20260926 22:40:26.864255 18422 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:40:26.864784 18422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: Bootstrap starting.
I20260926 22:40:26.867676 18422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:40:26.867909 18422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: Bootstrap complete.
I20260926 22:40:26.868099 18422 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true } }
I20260926 22:40:26.868196 18422 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab6c05c274ba4e3989741959ee16ed82, State: Initialized, Role: LEARNER
I20260926 22:40:26.868237 18422 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true } }
I20260926 22:40:26.868402 18422 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:26.868410 18425 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true } } }
I20260926 22:40:26.868575 18425 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: This master's current role is: LEARNER
I20260926 22:40:26.869925 18436 catalog_manager.cc:1259] Loaded cluster ID: c4f1ae9664b646ac9c393356187e9503
I20260926 22:40:26.869974 18436 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: loading cluster ID for follower catalog manager: success
I20260926 22:40:26.870854 18436 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: acquiring CA information for follower catalog manager: success
I20260926 22:40:26.871132 18436 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:40:26.917963 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:26.910689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.11.107.253:36977)
0926 22:40:26.910937 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:26.910941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:26.910955 (+ 14us) server_negotiation.cc:407] Connection header received
0926 22:40:26.910985 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:26.910986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:26.910998 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:26.911875 (+ 877us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:26.911917 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.912698 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.913636 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:26.913895 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:26.913966 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:26.915428 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:26.915430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:26.915430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:26.915437 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:26.916965 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.917258 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.917258 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.917259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.917309 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:26.917466 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:26.917467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:26.917467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:26.917601 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:26.917679 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:26.917760 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 22:40:26.917819 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":56,"threads_started":1}
WARNING: no policy specified for oryx/127.11.107.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.107.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.107.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.252.keytab.
Entry for principal oryx/127.11.107.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.252.keytab.
I20260926 22:40:26.940824 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.252:35439
--webserver_interface=127.11.107.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.252.keytab
--principal=oryx/127.11.107.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:32867,127.11.107.253:36977,127.11.107.252:35439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:27.020576 18440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:27.020819 18440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:27.020856 18440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:27.022084 18440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:27.022147 18440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:27.022172 18440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:27.022213 18440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:27.022243 18440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:27.023703 18440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:32867,127.11.107.253:36977,127.11.107.252:35439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.252.keytab
--principal=oryx/127.11.107.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.252:35439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.107.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:27.023974 18440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:27.024196 18440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:27.028975 18448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:27.028970 18446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:27.028970 18445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:27.029023 18440 server_base.cc:1034] running on GCE node
I20260926 22:40:27.029444 18440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:27.029762 18440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:27.030923 18440 hybrid_clock.cc:648] HybridClock initialized: now 1790462427030906 us; error 30 us; skew 500 ppm
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:27.033347 18440 init.cc:377] Logged in from keytab as oryx/127.11.107.252@KRBTEST.COM (short username oryx)
I20260926 22:40:27.033829 18440 webserver.cc:460] Webserver started at http://127.11.107.252:45947/ using document root <none> and password file <none>
I20260926 22:40:27.034137 18440 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:27.034220 18440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:27.034369 18440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:27.035504 18440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/data/instance:
uuid: "b7a1144270434ce398847d0d7b7c144b"
format_stamp: "Formatted at 2026-09-26 22:40:27 on dist-test-slave-j8sq"
server_key: "3fd393467676b564561635b8acf244cd"
server_key_iv: "ec76ade208ea470ff0a1c70ee4df0b3c"
server_key_version: "encryptionkey@0"
I20260926 22:40:27.035892 18440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/wal/instance:
uuid: "b7a1144270434ce398847d0d7b7c144b"
format_stamp: "Formatted at 2026-09-26 22:40:27 on dist-test-slave-j8sq"
server_key: "3fd393467676b564561635b8acf244cd"
server_key_iv: "ec76ade208ea470ff0a1c70ee4df0b3c"
server_key_version: "encryptionkey@0"
I20260926 22:40:27.037212 18440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:27.037941 18455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.038107 18440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:27.038211 18440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/wal
uuid: "b7a1144270434ce398847d0d7b7c144b"
format_stamp: "Formatted at 2026-09-26 22:40:27 on dist-test-slave-j8sq"
server_key: "3fd393467676b564561635b8acf244cd"
server_key_iv: "ec76ade208ea470ff0a1c70ee4df0b3c"
server_key_version: "encryptionkey@0"
I20260926 22:40:27.038291 18440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:27.054319 18440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:27.056432 18440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:27.056617 18440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:27.061106 18440 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.252:35439
I20260926 22:40:27.061177 18507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.252:35439 every 8 connection(s)
I20260926 22:40:27.061579 18440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/data/info.pb
I20260926 22:40:27.062605 18508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:27.064066 18508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:40:27.065097 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 18440
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.252@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.107.252@KRBTEST.COM
I20260926 22:40:27.074272 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.064842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37732 (local address 127.11.107.254:32867)
0926 22:40:27.064912 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.064914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.065002 (+ 88us) server_negotiation.cc:407] Connection header received
0926 22:40:27.065575 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.065577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.065591 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.065653 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.066332 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.066935 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.068112 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.068445 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.068506 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.071022 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.071024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.071024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.071032 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.072552 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.073138 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.073139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.073140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.073215 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.073503 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.073504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.073504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.073633 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.073677 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.074088 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.074179 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 22:40:27.074487 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.064686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:37732)
0926 22:40:27.064860 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.064902 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.065448 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.065755 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.065762 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.066083 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.066237 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.066242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.067026 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.067027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.068035 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.068038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.068538 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.068539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.068668 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.069262 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.069269 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.070941 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.072637 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.072643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.072646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.073047 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.073289 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.073290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.073290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.073415 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.073729 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.073734 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.073928 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.074044 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.074263 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":107,"thread_start_us":99,"threads_started":1}
I20260926 22:40:27.075165 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.065586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.11.107.252:35439)
0926 22:40:27.065756 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.065760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.065781 (+ 21us) server_negotiation.cc:407] Connection header received
0926 22:40:27.065809 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.065811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.065829 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.065943 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.066580 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.067632 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.068459 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.068912 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.068981 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.071698 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.071704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.071704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.071710 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.073403 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.073773 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.073776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.073776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.073824 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.074115 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.074116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.074116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.074335 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.074931 (+ 596us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.074983 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.075055 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260926 22:40:27.075848 18508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.252@KRBTEST.COM for oryx/127.11.107.253@KRBTEST.COM
I20260926 22:40:27.081598 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.076133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:36977 (local address 127.0.0.1:36294)
0926 22:40:27.076195 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.076207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.076250 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.076345 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.076346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.076450 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.076483 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.076484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.077058 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.077058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.077560 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.077561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.077940 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.077940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.077984 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.078407 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.078411 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.079533 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.080869 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.080870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.080871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.081017 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.081183 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.081184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.081184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.081229 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.081398 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.081398 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.081430 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.081477 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.081540 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 22:40:27.081692 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.076214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36294 (local address 127.11.107.253:36977)
0926 22:40:27.076254 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.076256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.076272 (+ 16us) server_negotiation.cc:407] Connection header received
0926 22:40:27.076292 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.076293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.076304 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.076343 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.076523 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.076998 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.077629 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.077827 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.077934 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.079608 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.079609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.079609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.079615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.080803 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.081078 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.081079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.081079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.081123 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.081266 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.081266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.081266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.081351 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.081394 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.081492 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.081551 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:40:27.082041 18508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.252@KRBTEST.COM for oryx/127.11.107.252@KRBTEST.COM
I20260926 22:40:27.087750 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.082334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:35439 (local address 127.0.0.1:57766)
0926 22:40:27.082388 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.082398 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.082443 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.082547 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.082547 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.082674 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.082725 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.082727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.083305 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.083306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.083868 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.083870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.084170 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.084170 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.084203 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.084494 (+ 291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.084498 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.085541 (+ 1043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.087023 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.087024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.087024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.087176 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.087333 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.087334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.087334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.087381 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.087535 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.087535 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.087563 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.087603 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.087674 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 22:40:27.087795 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.082386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.11.107.252:35439)
0926 22:40:27.082439 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.082440 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.082447 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:27.082480 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.082481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.082488 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.082529 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.082788 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.083250 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.083931 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.084114 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.084168 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.085607 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.085608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.085608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.085614 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.086959 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.087235 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.087236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.087236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.087279 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.087426 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.087426 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.087426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.087502 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.087543 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.087625 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.087707 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":38}
I20260926 22:40:27.089097 18508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: Bootstrap starting.
I20260926 22:40:27.089905 18508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:27.090211 18508 log.cc:824] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:27.090893 18508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: No bootstrap required, opened a new log
I20260926 22:40:27.092199 18508 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } }
I20260926 22:40:27.092329 18508 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:27.092351 18508 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a1144270434ce398847d0d7b7c144b, State: Initialized, Role: FOLLOWER
I20260926 22:40:27.092433 18508 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } }
I20260926 22:40:27.092656 18513 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } } }
I20260926 22:40:27.092712 18508 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:27.092738 18513 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: This master's current role is: FOLLOWER
W20260926 22:40:27.094164 18524 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:40:27.094219 18524 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:40:27.096393 18440 master_runner.cc:428] Detected that this master b7a1144270434ce398847d0d7b7c144b is joining an existing cluster
I20260926 22:40:27.096429 18440 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.107.252:35439
I20260926 22:40:27.103919 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.097383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.11.107.254:32867)
0926 22:40:27.097430 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.097432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.097567 (+ 135us) server_negotiation.cc:407] Connection header received
0926 22:40:27.097684 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.097684 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.097697 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.097731 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.098189 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.098963 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.100068 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.100358 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.100392 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.101407 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.101409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.101409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.101417 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.102913 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.103212 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.103213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.103214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.103273 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.103440 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.103441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.103442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.103565 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.103631 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.103760 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.103830 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:40:27.104055 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.097370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:37740)
0926 22:40:27.097506 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.097516 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.097592 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.097799 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.097800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.098024 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.098099 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.098101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.099045 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.099046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.099979 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.099981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.100488 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.100489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.100550 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.101004 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.101009 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.101312 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.102983 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.102984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.102984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.103138 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.103337 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.103337 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.103337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.103383 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.103621 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.103622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.103659 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.103715 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.103782 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":44,"thread_start_us":79,"threads_started":1}
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462426, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.254@KRBTEST.COM for oryx/127.11.107.252@KRBTEST.COM
I20260926 22:40:27.122186 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.097417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.11.107.252:35439)
0926 22:40:27.097504 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.097505 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.097788 (+ 283us) server_negotiation.cc:407] Connection header received
0926 22:40:27.097895 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.097895 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.097903 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.097927 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.098277 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.098956 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.100067 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.100351 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.100389 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.119197 (+ 18808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.119199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.119200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.119207 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.120834 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.121363 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.121364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.121365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.121411 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.121590 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.121590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.121591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.121729 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.121772 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.121975 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.122056 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 22:40:27.122491 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.097665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.11.107.253:36977)
0926 22:40:27.097707 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.097708 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.098009 (+ 301us) server_negotiation.cc:407] Connection header received
0926 22:40:27.098101 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.098102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.098111 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.098139 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.098524 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.099259 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.100335 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.100655 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.100704 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.119575 (+ 18871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.119577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.119577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.119586 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.121182 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.121654 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.121655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.121655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.121702 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.121920 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.121921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.121921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.122048 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.122087 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.122305 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.122403 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 22:40:27.122772 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.097467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:35439 (local address 127.0.0.1:57774)
0926 22:40:27.097739 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.097748 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.097805 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.097974 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.097974 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.098152 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.098209 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.098210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.099040 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.099040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.099979 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.099981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.100487 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.100487 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.100548 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.118455 (+ 17907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.118462 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.119067 (+ 605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.120926 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.120928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.120928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.121282 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.121477 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.121477 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.121478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.121526 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.121811 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.121812 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.121872 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.122439 (+ 567us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.122648 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":37,"thread_start_us":69,"threads_started":1}
I20260926 22:40:27.122838 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.105955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:35439 (local address 127.0.0.1:57790)
0926 22:40:27.106005 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.106015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.106049 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.106267 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.106268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.106414 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.106444 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.106446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.107068 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.107069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.107561 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.107563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.107935 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.107935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.107997 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.108373 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.108378 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.109499 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.121541 (+ 12042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.121542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.121543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.121783 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.122028 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.122029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.122029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.122087 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.122509 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.122510 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.122560 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.122607 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.122731 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 22:40:27.122852 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.097619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:36977 (local address 127.0.0.1:36304)
0926 22:40:27.097947 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.097958 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.098033 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.098199 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.098200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.098379 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.098435 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.098437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.099369 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.099370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.100238 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.100241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.100777 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.100778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.100841 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.118763 (+ 17922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.118769 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.119448 (+ 679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.121277 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.121278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.121278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.121562 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.121775 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.121776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.121776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.121826 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.122133 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.122134 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.122183 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.122578 (+ 395us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.122703 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":26,"thread_start_us":55,"threads_started":1}
I20260926 22:40:27.123757 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.106012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.11.107.252:35439)
0926 22:40:27.106152 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.106158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.106163 (+ 5us) server_negotiation.cc:407] Connection header received
0926 22:40:27.106189 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.106189 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.106196 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.106265 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.106523 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.106991 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.107659 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.107864 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.107936 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.109592 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.109594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.109594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.109600 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.121440 (+ 11840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.121898 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.121899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.121899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.121953 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.122153 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.122154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.122154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.122287 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.123514 (+ 1227us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.123551 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.123631 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260926 22:40:27.124178 17788 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.107.252:35439: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b7a1144270434ce398847d0d7b7c144b"
member_type: NON_VOTER
last_known_addr {
host: "127.11.107.252"
port: 35439
}
attrs {
promote: true
}
}
dest_uuid: "ce9c66c085544744a0b3ba6679cd761e"
cas_config_opid_index: 5
I20260926 22:40:27.124442 17788 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } }
I20260926 22:40:27.124818 18347 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b7a1144270434ce398847d0d7b7c144b (127.11.107.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } } }
I20260926 22:40:27.125015 18482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:27.125771 18347 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.107.252:35439
I20260926 22:40:27.125876 18347 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } } }
I20260926 22:40:27.125942 18347 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:27.126021 18482 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 1 FOLLOWER]: Refusing update from remote peer ce9c66c085544744a0b3ba6679cd761e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 22:40:27.126204 18440 master.cc:537] Master@127.11.107.252:35439 shutting down...
I20260926 22:40:27.127031 18348 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 22:40:27.127970 17773 consensus_peers.cc:487] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e -> Peer b7a1144270434ce398847d0d7b7c144b (127.11.107.252:35439): Couldn't send request to peer b7a1144270434ce398847d0d7b7c144b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 22:40:27.129119 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.124886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:36977 (local address 127.0.0.1:36314)
0926 22:40:27.125123 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.125135 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.125182 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.125386 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.125387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.125389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:27.125433 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.125434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.126402 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.126403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.127587 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.127590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.128849 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.128850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.128924 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.128947 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.128997 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.129038 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260926 22:40:27.129120 18440 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:27.129200 18440 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:27.129241 18440 tablet_replica.cc:330] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: stopping tablet replica
I20260926 22:40:27.129540 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.124957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.11.107.253:36977)
0926 22:40:27.125064 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.125065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.125234 (+ 169us) server_negotiation.cc:407] Connection header received
0926 22:40:27.125255 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.125256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.125278 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.125332 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:27.125498 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.126229 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.127815 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.128444 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.128507 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.128585 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.129422 (+ 837us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.129465 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 22:40:27.129740 18396 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 LEARNER]: Refusing update from remote peer ce9c66c085544744a0b3ba6679cd761e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 22:40:27.129963 18348 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260926 22:40:27.130153 18456 connection.cc:441] server connection from 127.0.0.1:57762 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:57762 (request call id 19) could send its response
I20260926 22:40:27.130501 18396 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b7a1144270434ce398847d0d7b7c144b (127.11.107.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } } }
I20260926 22:40:27.131613 18535 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } } }
I20260926 22:40:27.131726 18535 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: This master's current role is: LEARNER
I20260926 22:40:27.131886 18425 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce9c66c085544744a0b3ba6679cd761e. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: NON_VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: true } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } } }
I20260926 22:40:27.131965 18425 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: This master's current role is: LEARNER
I20260926 22:40:27.131951 18348 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e: attempting to promote NON_VOTER ab6c05c274ba4e3989741959ee16ed82 to VOTER
I20260926 22:40:27.132133 18348 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } }
I20260926 22:40:27.132593 18396 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 LEARNER]: Refusing update from remote peer ce9c66c085544744a0b3ba6679cd761e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 22:40:27.132665 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.132554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:35439 (local address 127.0.0.1:57792)
0926 22:40:27.132586 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.132629 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 22:40:27.132828 18347 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 22:40:27.132938 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.132864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:35439 (local address 127.0.0.1:57796)
0926 22:40:27.132891 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.132916 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 22:40:27.133002 17773 consensus_peers.cc:487] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e -> Peer b7a1144270434ce398847d0d7b7c144b (127.11.107.252:35439): Couldn't send request to peer b7a1144270434ce398847d0d7b7c144b. Status: Network error: Client connection negotiation failed: client connection to 127.11.107.252:35439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 22:40:27.133507 18536 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ab6c05c274ba4e3989741959ee16ed82 (127.11.107.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } } }
I20260926 22:40:27.133682 18396 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ab6c05c274ba4e3989741959ee16ed82 (127.11.107.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } } }
I20260926 22:40:27.134521 18425 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } } }
I20260926 22:40:27.134608 18425 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:40:27.134574 18348 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } } }
I20260926 22:40:27.134670 18348 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:27.134766 18536 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } } }
I20260926 22:40:27.134819 18536 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:27.142180 18440 master.cc:552] Master@127.11.107.252:35439 shutdown complete.
I20260926 22:40:27.142241 18440 master_runner.cc:305] Clearing existing system tablet
I20260926 22:40:27.142364 18440 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:27.142419 18440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:27.143023 18440 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:27.143503 18538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.143661 18440 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:27.143719 18440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:27.143764 18440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/wal
uuid: "b7a1144270434ce398847d0d7b7c144b"
format_stamp: "Formatted at 2026-09-26 22:40:27 on dist-test-slave-j8sq"
server_key: "3fd393467676b564561635b8acf244cd"
server_key_iv: "ec76ade208ea470ff0a1c70ee4df0b3c"
server_key_version: "encryptionkey@0"
I20260926 22:40:27.144344 18440 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:27.145503 18440 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 22:40:27.145558 18440 log.cc:1196] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 22:40:27.145754 18440 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: Deleting consensus metadata
I20260926 22:40:27.145900 18440 master_runner.cc:315] Copying system tablet from 127.11.107.254:32867
I20260926 22:40:27.146539 18440 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: tablet copy: Beginning tablet copy session from remote peer at address 127.11.107.254:32867
I20260926 22:40:27.152961 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.146862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:37750)
0926 22:40:27.147008 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.147018 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.147075 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.147198 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.147199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.147346 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.147420 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.147422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.148359 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.148359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.148917 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.148919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.149382 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.149382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.149430 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.149924 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.149930 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.150231 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.151909 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.151910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.151910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.152075 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.152276 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.152277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.152277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.152325 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.152627 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.152629 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.152683 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.152763 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.152863 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260926 22:40:27.152984 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.146914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.11.107.254:32867)
0926 22:40:27.146956 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.146958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.147087 (+ 129us) server_negotiation.cc:407] Connection header received
0926 22:40:27.147117 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.147118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.147129 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.147196 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.147514 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.148285 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.149000 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.149305 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.149384 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.150310 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.150312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.150312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.150319 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.151825 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.152142 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.152143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.152144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.152207 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.152384 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.152385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.152385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.152520 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.152594 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.152807 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.152889 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:40:27.153577 17808 tablet_copy_service.cc:140] P ce9c66c085544744a0b3ba6679cd761e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b7a1144270434ce398847d0d7b7c144b ({username='oryx', principal='oryx/127.11.107.252@KRBTEST.COM'} at 127.0.0.1:37750)
I20260926 22:40:27.153666 17808 tablet_copy_service.cc:161] P ce9c66c085544744a0b3ba6679cd761e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b7a1144270434ce398847d0d7b7c144b at {username='oryx', principal='oryx/127.11.107.252@KRBTEST.COM'} at 127.0.0.1:37750: session id = b7a1144270434ce398847d0d7b7c144b-00000000000000000000000000000000
I20260926 22:40:27.154211 17808 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 22:40:27.154842 18440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 22:40:27.155928 18440 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: tablet copy: Starting download of 0 data blocks...
I20260926 22:40:27.156165 18440 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: tablet copy: Starting download of 1 WAL segments...
I20260926 22:40:27.157022 18440 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 22:40:27.158002 17808 tablet_copy_service.cc:342] P ce9c66c085544744a0b3ba6679cd761e: Request end of tablet copy session b7a1144270434ce398847d0d7b7c144b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.107.252@KRBTEST.COM'} at 127.0.0.1:37750
I20260926 22:40:27.158061 17808 tablet_copy_service.cc:433] P ce9c66c085544744a0b3ba6679cd761e: ending tablet copy session b7a1144270434ce398847d0d7b7c144b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b7a1144270434ce398847d0d7b7c144b
I20260926 22:40:27.160585 18440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:27.161806 18545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:27.161820 18546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:27.161850 18548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:27.162001 18440 server_base.cc:1034] running on GCE node
I20260926 22:40:27.162129 18440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:27.162271 18440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:27.163407 18440 hybrid_clock.cc:648] HybridClock initialized: now 1790462427163401 us; error 24 us; skew 500 ppm
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:27.164678 18440 init.cc:377] Logged in from keytab as oryx/127.11.107.252@KRBTEST.COM (short username oryx)
I20260926 22:40:27.164966 18440 webserver.cc:460] Webserver started at http://127.11.107.252:37133/ using document root <none> and password file <none>
I20260926 22:40:27.165095 18440 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:27.165129 18440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:27.165767 18440 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:27.166347 18555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.166464 18440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.166509 18440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/wal
uuid: "b7a1144270434ce398847d0d7b7c144b"
format_stamp: "Formatted at 2026-09-26 22:40:27 on dist-test-slave-j8sq"
server_key: "3fd393467676b564561635b8acf244cd"
server_key_iv: "ec76ade208ea470ff0a1c70ee4df0b3c"
server_key_version: "encryptionkey@0"
I20260926 22:40:27.166564 18440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:27.188758 18440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:27.190325 18440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:27.193399 18440 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.252:35439
I20260926 22:40:27.193626 18607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.252:35439 every 8 connection(s)
I20260926 22:40:27.193809 18440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/master-2/data/info.pb
I20260926 22:40:27.194741 18608 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:40:27.195287 18608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: Bootstrap starting.
I20260926 22:40:27.198441 18608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:40:27.198698 18608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: Bootstrap complete.
I20260926 22:40:27.198907 18608 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } }
I20260926 22:40:27.199003 18608 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7a1144270434ce398847d0d7b7c144b, State: Initialized, Role: LEARNER
I20260926 22:40:27.199043 18608 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } }
I20260926 22:40:27.199239 18608 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:27.199257 18611 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } } }
I20260926 22:40:27.199424 18611 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: This master's current role is: LEARNER
I20260926 22:40:27.200817 18622 catalog_manager.cc:1259] Loaded cluster ID: c4f1ae9664b646ac9c393356187e9503
I20260926 22:40:27.200992 18622 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: loading cluster ID for follower catalog manager: success
I20260926 22:40:27.202523 18622 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: acquiring CA information for follower catalog manager: success
I20260926 22:40:27.202845 18622 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:40:27.238644 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.231539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.11.107.252:35439)
0926 22:40:27.231658 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.231665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.231674 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:27.231697 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.231698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.231707 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.231784 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.232499 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.233159 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.234274 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.234479 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.234556 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.236072 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.236073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.236074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.236079 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.237597 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.238017 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.238018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.238019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.238068 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.238248 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.238248 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.238249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.238336 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.238403 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.238520 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.238576 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
I20260926 22:40:27.239202 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 17850
I20260926 22:40:27.244234 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:44109
--local_ip_for_outbound_sockets=127.11.107.193
--tserver_master_addrs=127.11.107.254:32867
--webserver_port=39411
--webserver_interface=127.11.107.193
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab
--principal=oryx/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.107.254:32867,127.11.107.253:36977
--tserver_master_addrs=127.11.107.254:32867,127.11.107.253:36977,127.11.107.252:35439 with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:27.328114 18624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:27.328320 18624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:27.328357 18624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:27.329423 18624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:27.329463 18624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:27.329500 18624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:27.330983 18624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab
--principal=oryx/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:44109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=39411
--tserver_master_addrs=127.11.107.254:32867,127.11.107.253:36977,127.11.107.252:35439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:27.331178 18624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:27.331359 18624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:27.336349 18630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:27.336333 18632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:27.336431 18624 server_base.cc:1034] running on GCE node
W20260926 22:40:27.336333 18629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:27.336730 18624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:27.336978 18624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:27.338140 18624 hybrid_clock.cc:648] HybridClock initialized: now 1790462427338134 us; error 38 us; skew 500 ppm
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:27.340551 18624 init.cc:377] Logged in from keytab as oryx/127.11.107.193@KRBTEST.COM (short username oryx)
I20260926 22:40:27.340911 18624 webserver.cc:460] Webserver started at http://127.11.107.193:39411/ using document root <none> and password file <none>
I20260926 22:40:27.341084 18624 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:27.341132 18624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:27.342423 18624 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:27.342938 18639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.343122 18624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.343186 18624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "5c80eb7981614c99afa3149d5f4cc225"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "43e1ade7a4d12bc582243096f40e424a"
server_key_iv: "48005147c4c028ad4fe551a09b0e23b8"
server_key_version: "encryptionkey@0"
I20260926 22:40:27.343469 18624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:27.368309 18624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:27.370867 18624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:27.371053 18624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:27.371335 18624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:27.371702 18624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:27.371769 18624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.371803 18624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:27.371829 18624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.377647 18624 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:44109
I20260926 22:40:27.377728 18752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:44109 every 8 connection(s)
I20260926 22:40:27.378103 18624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:27.379422 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 18624
I20260926 22:40:27.379576 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 17984
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
I20260926 22:40:27.387035 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:35919
--local_ip_for_outbound_sockets=127.11.107.194
--tserver_master_addrs=127.11.107.254:32867
--webserver_port=40725
--webserver_interface=127.11.107.194
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab
--principal=oryx/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.107.254:32867,127.11.107.253:36977
--tserver_master_addrs=127.11.107.254:32867,127.11.107.253:36977,127.11.107.252:35439 with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.252@KRBTEST.COM
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.253@KRBTEST.COM
I20260926 22:40:27.391100 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.378866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:58915 (local address 127.11.107.252:35439)
0926 22:40:27.378955 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.378957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.379200 (+ 243us) server_negotiation.cc:407] Connection header received
0926 22:40:27.380484 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.380485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.380499 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.380535 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.381654 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.382448 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.384022 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.384335 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.384370 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.387959 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.387960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.387960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.387970 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.389656 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.390135 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.390137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.390137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.390200 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.390451 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.390452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.390453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.390575 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.390652 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.390921 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.391007 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 22:40:27.391314 18757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.378798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:35439 (local address 127.11.107.193:58915)
0926 22:40:27.379055 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.379095 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.380356 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.380707 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.380711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.381216 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.381543 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.381550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.382583 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.382584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.383903 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.383905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.384451 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.384452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.384598 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.385273 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.385285 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.387864 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.389763 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.389770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.389772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.390061 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.390274 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.390275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.390276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.390384 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.390642 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.390648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.390792 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.390901 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.391064 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":83,"thread_start_us":81,"threads_started":1}
I20260926 22:40:27.391829 18753 heartbeater.cc:347] Connected to a master server at 127.11.107.252:35439
I20260926 22:40:27.391909 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.379069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:37559 (local address 127.11.107.253:36977)
0926 22:40:27.379114 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.379116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.379282 (+ 166us) server_negotiation.cc:407] Connection header received
0926 22:40:27.380485 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.380486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.380499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.380530 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.381656 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.382490 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.384019 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.384369 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.384414 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.388821 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.388822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.388823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.388831 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.390597 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.391052 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.391053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.391054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.391103 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.391293 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.391294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.391294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.391434 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.391476 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.391682 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.391770 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:40:27.391919 18753 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:27.392109 18753 heartbeater.cc:511] Master 127.11.107.252:35439 requested a full tablet report, sending...
I20260926 22:40:27.392454 18759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.378797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:36977 (local address 127.11.107.193:37559)
0926 22:40:27.379219 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.379229 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.380365 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.381061 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.381062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.381358 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.381542 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.381548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.382606 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.382606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.383906 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.383908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.384489 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.384490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.384599 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.385359 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.385365 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.388714 (+ 3349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.390697 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.390698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.390699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.390905 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.391173 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.391174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.391174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.391231 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.391518 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.391519 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.391561 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.392032 (+ 471us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.392141 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":113,"thread_start_us":70,"threads_started":1}
I20260926 22:40:27.392475 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.378818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:33593 (local address 127.11.107.254:32867)
0926 22:40:27.378917 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.378920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.379290 (+ 370us) server_negotiation.cc:407] Connection header received
0926 22:40:27.380491 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.380493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.380505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.380536 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.381654 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.382460 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.384023 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.384314 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.384357 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.388293 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.388294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.388294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.388301 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.389813 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.391630 (+ 1817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.391631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.391632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.391706 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.391915 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.391916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.391916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.392042 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.392085 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.392302 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.392389 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 22:40:27.392625 18572 ts_manager.cc:194] Registered new tserver with Master: 5c80eb7981614c99afa3149d5f4cc225 (127.11.107.193:44109)
I20260926 22:40:27.392766 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.378798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.193:33593)
0926 22:40:27.379229 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.379239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.380366 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.380708 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.380711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.381217 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.381543 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.381550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.382593 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.382593 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.383903 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.383905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.384453 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.384453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.384598 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.385304 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.385308 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.388172 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.390965 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.390966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.390966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.391544 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.391781 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.391782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.391782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.391842 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.392133 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.392134 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.392170 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.392550 (+ 380us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.392649 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":460,"thread_start_us":64,"threads_started":1}
I20260926 22:40:27.392874 18755 heartbeater.cc:347] Connected to a master server at 127.11.107.253:36977
I20260926 22:40:27.392918 18755 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:27.392992 18755 heartbeater.cc:511] Master 127.11.107.253:36977 requested a full tablet report, sending...
I20260926 22:40:27.393083 18754 heartbeater.cc:347] Connected to a master server at 127.11.107.254:32867
I20260926 22:40:27.393117 18754 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:27.393181 18754 heartbeater.cc:511] Master 127.11.107.254:32867 requested a full tablet report, sending...
I20260926 22:40:27.393373 18386 ts_manager.cc:194] Registered new tserver with Master: 5c80eb7981614c99afa3149d5f4cc225 (127.11.107.193:44109)
I20260926 22:40:27.393432 17788 ts_manager.cc:194] Re-registered known tserver with Master: 5c80eb7981614c99afa3149d5f4cc225 (127.11.107.193:44109)
I20260926 22:40:27.394006 17788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:33593
W20260926 22:40:27.470738 18760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:27.470925 18760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:27.470948 18760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:27.472010 18760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:27.472044 18760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:27.472069 18760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:27.473563 18760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab
--principal=oryx/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:35919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=40725
--tserver_master_addrs=127.11.107.254:32867,127.11.107.253:36977,127.11.107.252:35439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:27.473753 18760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:27.473948 18760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:27.478945 18766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:27.478971 18765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:27.478961 18768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:27.479163 18760 server_base.cc:1034] running on GCE node
I20260926 22:40:27.479388 18760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:27.479655 18760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:27.480799 18760 hybrid_clock.cc:648] HybridClock initialized: now 1790462427480792 us; error 28 us; skew 500 ppm
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:27.483372 18760 init.cc:377] Logged in from keytab as oryx/127.11.107.194@KRBTEST.COM (short username oryx)
I20260926 22:40:27.483842 18760 webserver.cc:460] Webserver started at http://127.11.107.194:40725/ using document root <none> and password file <none>
I20260926 22:40:27.484078 18760 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:27.484157 18760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:27.485419 18760 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:27.486020 18775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.486197 18760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:27.486291 18760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "fcf3b707309165b2be0384c89cd70592"
server_key_iv: "8cb3006888262c8127a8a8bd2188744d"
server_key_version: "encryptionkey@0"
I20260926 22:40:27.486632 18760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:27.504695 18760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:27.506459 18760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:27.506613 18760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:27.506830 18760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:27.507159 18760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:27.507213 18760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.507249 18760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:27.507274 18760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.512965 18760 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:35919
I20260926 22:40:27.513038 18888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:35919 every 8 connection(s)
I20260926 22:40:27.513412 18760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:40:27.513867 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 18760
I20260926 22:40:27.514015 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 18118
I20260926 22:40:27.521090 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:33873
--local_ip_for_outbound_sockets=127.11.107.195
--tserver_master_addrs=127.11.107.254:32867
--webserver_port=43501
--webserver_interface=127.11.107.195
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab
--principal=oryx/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.107.254:32867,127.11.107.253:36977
--tserver_master_addrs=127.11.107.254:32867,127.11.107.253:36977,127.11.107.252:35439 with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.252@KRBTEST.COM
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.253@KRBTEST.COM
I20260926 22:40:27.527232 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.514240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:35439 (local address 127.11.107.194:42771)
0926 22:40:27.518342 (+ 4102us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.518354 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.518428 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.518599 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.518600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.518833 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.518934 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.518936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.519852 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.519853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.520755 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.520757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.521581 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.521582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.521659 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.522110 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.522115 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.523759 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.525573 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.525574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.525575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.525881 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.526046 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.526047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.526048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.526195 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.526473 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.526480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.526683 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.526796 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.526963 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4071,"mutex_wait_us":130,"thread_start_us":87,"threads_started":1}
I20260926 22:40:27.527403 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.514646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:41767 (local address 127.11.107.254:32867)
0926 22:40:27.514699 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.514701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.514732 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:40:27.516216 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.516217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.516227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.516279 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.517165 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.517643 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.519299 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.519651 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.519699 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.522920 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.522921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.522921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.522929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.525282 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.526195 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.526196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.526197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.526257 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.526745 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.526746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.526746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.526898 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.526955 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.527181 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.527268 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:40:27.527942 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.514519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:42771 (local address 127.11.107.252:35439)
0926 22:40:27.514656 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.514659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.518417 (+ 3758us) server_negotiation.cc:407] Connection header received
0926 22:40:27.518481 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.518482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.518496 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.518547 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.519037 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.519764 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.520884 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.521187 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.521235 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.523851 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.523852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.523852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.523862 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.525487 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.525938 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.525939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.525940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.525996 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.526277 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.526278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.526278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.526420 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.526482 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.527740 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.527829 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 22:40:27.528455 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.518979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:57269 (local address 127.11.107.253:36977)
0926 22:40:27.519026 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.519027 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.520912 (+ 1885us) server_negotiation.cc:407] Connection header received
0926 22:40:27.521009 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.521010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.521022 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.521052 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.521460 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.522214 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.523182 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.523492 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.523539 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.525588 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.525589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.525589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.525597 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.527285 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.527660 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.527661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.527661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.527717 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.527919 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.527920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.527920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.528041 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.528087 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.528291 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.528363 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:40:27.528466 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.518800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:36977 (local address 127.11.107.194:57269)
0926 22:40:27.520842 (+ 2042us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.520854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.520936 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.521119 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.521119 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.521289 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.521364 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.521366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.522306 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.522307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.523084 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.523086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.523616 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.523617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.523702 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.524034 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.524040 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.525487 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.527384 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.527386 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.527386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.527586 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.527790 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.527791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.527791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.527848 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.528132 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.528132 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.528181 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.528253 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.528352 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2025,"thread_start_us":79,"threads_started":1}
I20260926 22:40:27.528491 18889 heartbeater.cc:347] Connected to a master server at 127.11.107.252:35439
I20260926 22:40:27.528584 18889 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:27.528785 18889 heartbeater.cc:511] Master 127.11.107.252:35439 requested a full tablet report, sending...
I20260926 22:40:27.529032 18891 heartbeater.cc:347] Connected to a master server at 127.11.107.253:36977
I20260926 22:40:27.529079 18891 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:27.529140 18891 heartbeater.cc:511] Master 127.11.107.253:36977 requested a full tablet report, sending...
I20260926 22:40:27.529256 18572 ts_manager.cc:194] Registered new tserver with Master: b0ce7f00f3424cf0982c2b4de5e5cdeb (127.11.107.194:35919)
I20260926 22:40:27.529567 18386 ts_manager.cc:194] Registered new tserver with Master: b0ce7f00f3424cf0982c2b4de5e5cdeb (127.11.107.194:35919)
I20260926 22:40:27.527786 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.514239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.194:41767)
0926 22:40:27.514608 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.514656 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.516091 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.516365 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.516369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.516786 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.517074 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.517081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.517748 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.517749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.519193 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.519196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.519787 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.519788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.519937 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.520686 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.520697 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.522812 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.525380 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.525387 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.525390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.526085 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.526333 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.526334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.526334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.526654 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.526992 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.526993 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.527040 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.527529 (+ 489us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.527639 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":32,"thread_start_us":83,"threads_started":1}
I20260926 22:40:27.530311 18890 heartbeater.cc:347] Connected to a master server at 127.11.107.254:32867
I20260926 22:40:27.530366 18890 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:27.530411 18890 heartbeater.cc:511] Master 127.11.107.254:32867 requested a full tablet report, sending...
I20260926 22:40:27.530627 17788 ts_manager.cc:194] Re-registered known tserver with Master: b0ce7f00f3424cf0982c2b4de5e5cdeb (127.11.107.194:35919)
I20260926 22:40:27.530944 17788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:41767
I20260926 22:40:27.590204 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.586786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:35439 (local address 127.0.0.1:57816)
0926 22:40:27.586944 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.586957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.587034 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.587202 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.587202 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.587205 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:27.587258 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.587260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.588285 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.588286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.589128 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.589132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.589977 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.589979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.590028 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.590032 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.590070 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.590104 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260926 22:40:27.590201 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.586965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.11.107.252:35439)
0926 22:40:27.587033 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.587035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.587046 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:40:27.587098 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.587099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.587124 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.587194 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:27.587349 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.588186 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.589210 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.589902 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.589980 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.590054 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.590075 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.590115 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 22:40:27.590580 18582 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 22:40:27.590762 18611 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce9c66c085544744a0b3ba6679cd761e. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } } }
I20260926 22:40:27.590876 18611 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: This master's current role is: LEARNER
W20260926 22:40:27.606206 18896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:27.606390 18896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:27.606412 18896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:27.607586 18896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:27.607645 18896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:27.607707 18896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:27.609082 18896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab
--principal=oryx/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:33873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=43501
--tserver_master_addrs=127.11.107.254:32867,127.11.107.253:36977,127.11.107.252:35439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:27.609314 18896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:27.609540 18896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:27.614343 18901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:27.614382 18904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:27.614375 18902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:27.614357 18896 server_base.cc:1034] running on GCE node
I20260926 22:40:27.614715 18896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:27.614943 18896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:27.616115 18896 hybrid_clock.cc:648] HybridClock initialized: now 1790462427616112 us; error 39 us; skew 500 ppm
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:27.618510 18896 init.cc:377] Logged in from keytab as oryx/127.11.107.195@KRBTEST.COM (short username oryx)
I20260926 22:40:27.618932 18896 webserver.cc:460] Webserver started at http://127.11.107.195:43501/ using document root <none> and password file <none>
I20260926 22:40:27.619167 18896 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:27.619249 18896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:27.620556 18896 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:27.621100 18911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.621261 18896 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:27.621348 18896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "7ea6064487bc48d7a050babd80852d04"
format_stamp: "Formatted at 2026-09-26 22:40:26 on dist-test-slave-j8sq"
server_key: "61c6cc8ec44d9c90531a21c2941d99ac"
server_key_iv: "58b71ac71b06c54cf14fad025fa07b39"
server_key_version: "encryptionkey@0"
I20260926 22:40:27.621665 18896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:27.633178 18896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:27.635291 18896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:27.635444 18896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:27.635674 18896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:27.635975 18896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:27.636029 18896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.636071 18896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:27.636099 18896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:27.641614 18896 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:33873
I20260926 22:40:27.641716 19024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:33873 every 8 connection(s)
I20260926 22:40:27.642059 18896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:40:27.646912 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 18896
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.253@KRBTEST.COM
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.252@KRBTEST.COM
Sep 26 22:40:27 dist-test-slave-j8sq krb5kdc[17743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462427, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
I20260926 22:40:27.653663 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.642844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:35439 (local address 127.11.107.195:42595)
0926 22:40:27.643007 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.643047 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.644075 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.644327 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.644330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.644651 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.644829 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.644834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.645781 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.645782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.646703 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.646704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.647264 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.647265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.647321 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.647787 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.647792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.650238 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.652020 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.652025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.652027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.652344 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.652536 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.652536 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.652537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.652662 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.652932 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.652939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.653104 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.653213 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.653392 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":59,"thread_start_us":90,"threads_started":1}
I20260926 22:40:27.654474 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.642853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:42595 (local address 127.11.107.252:35439)
0926 22:40:27.642899 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.642902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.643150 (+ 248us) server_negotiation.cc:407] Connection header received
0926 22:40:27.644180 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.644182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.644194 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.644224 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.644933 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.645684 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.646809 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.647117 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.647177 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.650337 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.650338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.650338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.650345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.651941 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.652419 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.652420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.652420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.652473 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.652733 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.652734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.652735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.652869 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.652930 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.654267 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.654357 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:40:27.655493 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.642844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:36977 (local address 127.11.107.195:43307)
0926 22:40:27.643007 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.643047 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.644057 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.644326 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.644329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.644636 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.644827 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.644831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.645511 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.645512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.646405 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.646408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.646852 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.646853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.647094 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.647656 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.647668 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.650589 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.653875 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.653876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.653876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.654083 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.654749 (+ 666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.654750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.654750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.654821 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.655151 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.655152 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.655208 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.655285 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.655376 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260926 22:40:27.655771 19025 heartbeater.cc:347] Connected to a master server at 127.11.107.252:35439
I20260926 22:40:27.655865 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.642848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:43307 (local address 127.11.107.253:36977)
0926 22:40:27.642898 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.642900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.643139 (+ 239us) server_negotiation.cc:407] Connection header received
0926 22:40:27.644180 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.644181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.644192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.644254 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.644925 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.645412 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.646499 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.646708 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.646783 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.650666 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.650667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.650667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.650674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.653777 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.654329 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.654330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.654331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.654394 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.654913 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.654914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.654914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.655054 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.655120 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.655666 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.655751 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 22:40:27.655874 19025 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:27.656051 19025 heartbeater.cc:511] Master 127.11.107.252:35439 requested a full tablet report, sending...
I20260926 22:40:27.656152 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.643143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.195:34433)
0926 22:40:27.643267 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.643277 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.644075 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.644327 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.644330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.644646 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:27.644829 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.644834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.645796 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.645797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.646698 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.646701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.647245 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.647246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.647795 (+ 549us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.648177 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:27.648182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:27.650821 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:27.654894 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.654894 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.654895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.655125 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.655352 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:27.655353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:27.655353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:27.655412 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:27.655687 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:27.655689 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:27.655731 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.655796 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.655886 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":25,"thread_start_us":64,"threads_started":1}
I20260926 22:40:27.656149 19027 heartbeater.cc:347] Connected to a master server at 127.11.107.253:36977
I20260926 22:40:27.656234 19027 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:27.656301 19027 heartbeater.cc:511] Master 127.11.107.253:36977 requested a full tablet report, sending...
I20260926 22:40:27.656478 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.643327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:34433 (local address 127.11.107.254:32867)
0926 22:40:27.643375 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.643377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.643384 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:27.644169 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.644170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.644181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.644215 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.644934 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.645695 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.646817 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.647104 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.647146 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.650912 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.650914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.650914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.650921 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.654172 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.655219 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.655220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.655221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.655282 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.655480 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.655481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.655481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.655597 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.655661 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.656324 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.656402 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:40:27.656481 18572 ts_manager.cc:194] Registered new tserver with Master: 7ea6064487bc48d7a050babd80852d04 (127.11.107.195:33873)
I20260926 22:40:27.656587 18386 ts_manager.cc:194] Registered new tserver with Master: 7ea6064487bc48d7a050babd80852d04 (127.11.107.195:33873)
I20260926 22:40:27.656759 19026 heartbeater.cc:347] Connected to a master server at 127.11.107.254:32867
I20260926 22:40:27.656797 19026 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:27.656857 19026 heartbeater.cc:511] Master 127.11.107.254:32867 requested a full tablet report, sending...
I20260926 22:40:27.657058 17788 ts_manager.cc:194] Re-registered known tserver with Master: 7ea6064487bc48d7a050babd80852d04 (127.11.107.195:33873)
I20260926 22:40:27.657563 17788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:34433
I20260926 22:40:27.658218 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.648999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36322 (local address 127.11.107.253:36977)
0926 22:40:27.649114 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.649116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.649343 (+ 227us) server_negotiation.cc:407] Connection header received
0926 22:40:27.649442 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.649443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.649455 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.649492 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.650515 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.651268 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.652236 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.652556 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.652594 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.654766 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.654767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.654767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.654774 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.656981 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.657379 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.657380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.657380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.657434 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.657625 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.657626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.657626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.657756 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.657827 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.658061 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.658134 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260926 22:40:27.660838 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.648686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.11.107.254:32867)
0926 22:40:27.649279 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.649281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.649289 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:40:27.649310 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.649311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.649323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.650591 (+ 1268us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.650629 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.652561 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.653719 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.654057 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.654468 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.657585 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.657586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.657586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.657593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.659737 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.660076 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.660077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.660077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.660128 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.660338 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.660339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.660339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.660446 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.660558 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.660650 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.660723 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":48,"threads_started":1}
I20260926 22:40:27.661715 19036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.648467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.11.107.252:35439)
0926 22:40:27.648906 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.648908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.648915 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:27.648936 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.648937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.648949 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.649006 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.649907 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.656570 (+ 6663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.657595 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.657945 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.658004 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.659220 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.659221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.659221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.659227 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.660571 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.660922 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.660923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.660923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.660982 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.661152 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.661152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.661153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.661278 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.661382 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.661504 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.661579 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":55,"threads_started":1}
I20260926 22:40:27.662387 17788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:27.663888 17788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:27.664705 18536 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: NON_VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: true } } }
I20260926 22:40:27.664806 18536 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:27.665877 19044 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e: attempting to promote NON_VOTER b7a1144270434ce398847d0d7b7c144b to VOTER
I20260926 22:40:27.665985 19044 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } }
I20260926 22:40:27.666460 18396 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 FOLLOWER]: Refusing update from remote peer ce9c66c085544744a0b3ba6679cd761e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 22:40:27.666653 18536 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 22:40:27.667021 18582 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 1 LEARNER]: Refusing update from remote peer ce9c66c085544744a0b3ba6679cd761e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 22:40:27.667200 19043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 22:40:27.667385 18536 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b7a1144270434ce398847d0d7b7c144b (127.11.107.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } } }
I20260926 22:40:27.667505 18396 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b7a1144270434ce398847d0d7b7c144b (127.11.107.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } } }
I20260926 22:40:27.668069 18582 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b7a1144270434ce398847d0d7b7c144b (127.11.107.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } } }
I20260926 22:40:27.668308 18536 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } } }
I20260926 22:40:27.668390 18536 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:27.668499 18536 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } } }
I20260926 22:40:27.668563 18536 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:27.668385 19052 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } } }
I20260926 22:40:27.668839 19052 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:27.668956 19053 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } } }
I20260926 22:40:27.669040 19053 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:40:27.669483 18611 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce9c66c085544744a0b3ba6679cd761e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } } }
I20260926 22:40:27.669575 18611 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:40:27.673903 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.669486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:44109 (local address 127.0.0.1:36724)
0926 22:40:27.669530 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.669541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.669579 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.670058 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.670059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.670061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:27.670106 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.670107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.671495 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.671497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.672687 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.672689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.673670 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.673671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.673733 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.673737 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.673765 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.673806 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 22:40:27.674288 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.669714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.11.107.193:44109)
0926 22:40:27.669882 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.669887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.669898 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:40:27.669938 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.669941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.669958 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.670048 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:27.670447 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.671415 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.672785 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.673574 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.674046 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.674127 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.674155 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.674206 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260926 22:40:27.674714 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.669718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.11.107.195:33873)
0926 22:40:27.669881 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.669886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.670602 (+ 716us) server_negotiation.cc:407] Connection header received
0926 22:40:27.670729 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.670732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.670747 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.670799 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:27.670988 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.671755 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.673319 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.674052 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.674119 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.674162 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.674594 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.674638 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260926 22:40:27.674902 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.669520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:33873 (local address 127.0.0.1:38102)
0926 22:40:27.670536 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.670546 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.670630 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.670875 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.670875 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.670877 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:27.670914 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.670915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.671850 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.671851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.673230 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.673232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.674220 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.674221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.674280 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.674284 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.674787 (+ 503us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.674832 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260926 22:40:27.675084 18687 tablet_service.cc:1467] Processing CreateTablet for tablet 7d31318e05904afd9f2d485eed9d2991 (DEFAULT_TABLE table=test-table [id=3019ccaba86e4e13acbb907b430105e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:27.675503 18687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d31318e05904afd9f2d485eed9d2991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:27.675753 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.670091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35919 (local address 127.0.0.1:34592)
0926 22:40:27.670560 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.670571 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.670613 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.671690 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.671691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.671693 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:27.671733 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.671735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.673023 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.673024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.674463 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.674466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.675497 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.675497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.675577 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.675582 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.675629 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.675677 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":69,"threads_started":1}
I20260926 22:40:27.675983 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.670289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.11.107.194:35919)
0926 22:40:27.671541 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.671545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.671552 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:27.671581 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.671582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.671594 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.671682 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:27.671786 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.672914 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.674559 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.675320 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.675405 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.675459 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.675854 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.675902 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1220,"thread_start_us":78,"threads_started":1}
I20260926 22:40:27.676626 18823 tablet_service.cc:1467] Processing CreateTablet for tablet 7d31318e05904afd9f2d485eed9d2991 (DEFAULT_TABLE table=test-table [id=3019ccaba86e4e13acbb907b430105e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:27.676959 18823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d31318e05904afd9f2d485eed9d2991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:27.679494 19059 tablet_bootstrap.cc:492] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225: Bootstrap starting.
I20260926 22:40:27.680341 19059 tablet_bootstrap.cc:654] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:27.680235 18959 tablet_service.cc:1467] Processing CreateTablet for tablet 7d31318e05904afd9f2d485eed9d2991 (DEFAULT_TABLE table=test-table [id=3019ccaba86e4e13acbb907b430105e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:27.680557 18959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d31318e05904afd9f2d485eed9d2991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:27.680454 19060 tablet_bootstrap.cc:492] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Bootstrap starting.
I20260926 22:40:27.680840 19059 log.cc:824] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:27.681020 19060 tablet_bootstrap.cc:654] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:27.681348 19060 log.cc:824] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:27.681804 19059 tablet_bootstrap.cc:492] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225: No bootstrap required, opened a new log
I20260926 22:40:27.681970 19059 ts_tablet_manager.cc:1397] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 22:40:27.682114 19060 tablet_bootstrap.cc:492] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb: No bootstrap required, opened a new log
I20260926 22:40:27.682196 19060 ts_tablet_manager.cc:1397] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:27.683904 19063 tablet_bootstrap.cc:492] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04: Bootstrap starting.
I20260926 22:40:27.684279 19060 raft_consensus.cc:348] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } }
I20260926 22:40:27.684451 19060 raft_consensus.cc:374] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:27.684501 19060 raft_consensus.cc:729] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ce7f00f3424cf0982c2b4de5e5cdeb, State: Initialized, Role: FOLLOWER
I20260926 22:40:27.684502 19063 tablet_bootstrap.cc:654] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:27.684644 19060 consensus_queue.cc:260] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } }
I20260926 22:40:27.684813 19063 log.cc:824] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:27.684911 19060 ts_tablet_manager.cc:1428] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 22:40:27.684854 19059 raft_consensus.cc:348] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } }
I20260926 22:40:27.685029 19059 raft_consensus.cc:374] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:27.685065 19059 raft_consensus.cc:729] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c80eb7981614c99afa3149d5f4cc225, State: Initialized, Role: FOLLOWER
I20260926 22:40:27.685169 19059 consensus_queue.cc:260] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } }
I20260926 22:40:27.685340 18890 heartbeater.cc:503] Master 127.11.107.254:32867 was elected leader, sending a full tablet report...
I20260926 22:40:27.685698 19059 ts_tablet_manager.cc:1428] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 22:40:27.685904 18754 heartbeater.cc:503] Master 127.11.107.254:32867 was elected leader, sending a full tablet report...
I20260926 22:40:27.686445 19063 tablet_bootstrap.cc:492] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04: No bootstrap required, opened a new log
I20260926 22:40:27.686508 19063 ts_tablet_manager.cc:1397] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 22:40:27.688534 19063 raft_consensus.cc:348] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } }
I20260926 22:40:27.688669 19063 raft_consensus.cc:374] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:27.688712 19063 raft_consensus.cc:729] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ea6064487bc48d7a050babd80852d04, State: Initialized, Role: FOLLOWER
I20260926 22:40:27.688814 19063 consensus_queue.cc:260] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } }
I20260926 22:40:27.689071 19063 ts_tablet_manager.cc:1428] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:40:27.689090 19026 heartbeater.cc:503] Master 127.11.107.254:32867 was elected leader, sending a full tablet report...
W20260926 22:40:27.768800 18892 tablet.cc:2369] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:40:27.806560 19067 raft_consensus.cc:482] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:27.806679 19067 raft_consensus.cc:504] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } }
I20260926 22:40:27.807088 19067 leader_election.cc:288] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c80eb7981614c99afa3149d5f4cc225 (127.11.107.193:44109), b0ce7f00f3424cf0982c2b4de5e5cdeb (127.11.107.194:35919)
I20260926 22:40:27.809867 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.807289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35919 (local address 127.11.107.195:58411)
0926 22:40:27.807348 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.807360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.807407 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.807619 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.807620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.807622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:27.807672 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.807673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.808327 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.808328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.809062 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.809064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.809590 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.809591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.809668 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.809672 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.809714 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.809744 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":1}
I20260926 22:40:27.809881 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.807410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:58411 (local address 127.11.107.194:35919)
0926 22:40:27.807464 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.807466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.807476 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:40:27.807503 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.807504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.807519 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.807590 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:27.807775 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.808217 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.809153 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.809508 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.809562 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.809569 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.809742 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.809789 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 22:40:27.809935 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.807395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:33469 (local address 127.11.107.193:44109)
0926 22:40:27.807440 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.807442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.807467 (+ 25us) server_negotiation.cc:407] Connection header received
0926 22:40:27.807502 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.807503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.807510 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.807548 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:27.807733 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.808210 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.809194 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.809533 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.809624 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.809632 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.809802 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.809864 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:40:27.809993 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.807300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:44109 (local address 127.11.107.195:33469)
0926 22:40:27.807361 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:27.807374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:27.807436 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:27.807577 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:27.807577 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:27.807579 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:27.807662 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.807664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.808296 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.808297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.809077 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:27.809079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.809645 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.809646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:27.809723 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.809727 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:27.809783 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 22:40:27.809816 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":52}
I20260926 22:40:27.810223 18843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d31318e05904afd9f2d485eed9d2991" candidate_uuid: "7ea6064487bc48d7a050babd80852d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" is_pre_election: true
I20260926 22:40:27.810352 18843 raft_consensus.cc:2454] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ea6064487bc48d7a050babd80852d04 in term 0.
I20260926 22:40:27.810329 18707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d31318e05904afd9f2d485eed9d2991" candidate_uuid: "7ea6064487bc48d7a050babd80852d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c80eb7981614c99afa3149d5f4cc225" is_pre_election: true
I20260926 22:40:27.810484 18707 raft_consensus.cc:2454] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ea6064487bc48d7a050babd80852d04 in term 0.
I20260926 22:40:27.810604 18913 leader_election.cc:302] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ea6064487bc48d7a050babd80852d04, b0ce7f00f3424cf0982c2b4de5e5cdeb; no voters:
I20260926 22:40:27.810773 19067 raft_consensus.cc:2781] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:27.810855 19067 raft_consensus.cc:482] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:27.810873 19067 raft_consensus.cc:3037] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:27.811565 19067 raft_consensus.cc:504] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } }
I20260926 22:40:27.811728 19067 leader_election.cc:288] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [CANDIDATE]: Term 1 election: Requested vote from peers 5c80eb7981614c99afa3149d5f4cc225 (127.11.107.193:44109), b0ce7f00f3424cf0982c2b4de5e5cdeb (127.11.107.194:35919)
I20260926 22:40:27.811825 18707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d31318e05904afd9f2d485eed9d2991" candidate_uuid: "7ea6064487bc48d7a050babd80852d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c80eb7981614c99afa3149d5f4cc225"
I20260926 22:40:27.811901 18707 raft_consensus.cc:3037] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:27.811990 18843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d31318e05904afd9f2d485eed9d2991" candidate_uuid: "7ea6064487bc48d7a050babd80852d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb"
I20260926 22:40:27.812071 18843 raft_consensus.cc:3037] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:27.812827 18707 raft_consensus.cc:2454] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ea6064487bc48d7a050babd80852d04 in term 1.
I20260926 22:40:27.812947 18843 raft_consensus.cc:2454] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ea6064487bc48d7a050babd80852d04 in term 1.
I20260926 22:40:27.812999 18912 leader_election.cc:302] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c80eb7981614c99afa3149d5f4cc225, 7ea6064487bc48d7a050babd80852d04; no voters:
I20260926 22:40:27.813093 19067 raft_consensus.cc:2781] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:27.813254 19067 raft_consensus.cc:686] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [term 1 LEADER]: Becoming Leader. State: Replica: 7ea6064487bc48d7a050babd80852d04, State: Running, Role: LEADER
I20260926 22:40:27.813364 19067 consensus_queue.cc:237] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } }
I20260926 22:40:27.814136 17787 catalog_manager.cc:5495] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ea6064487bc48d7a050babd80852d04 (127.11.107.195). New cstate: current_term: 1 leader_uuid: "7ea6064487bc48d7a050babd80852d04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20260926 22:40:27.852447 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.849936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38118 (local address 127.11.107.195:33873)
0926 22:40:27.849992 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.849994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.850012 (+ 18us) server_negotiation.cc:407] Connection header received
0926 22:40:27.850054 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.850055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.850071 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.850147 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:27.850297 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.850781 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.851532 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.851731 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.851817 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.851993 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:27.852098 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:27.852199 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.852370 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.852391 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:40:27.854219 18707 raft_consensus.cc:1261] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225 [term 1 FOLLOWER]: Refusing update from remote peer 7ea6064487bc48d7a050babd80852d04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:27.854243 18843 raft_consensus.cc:1261] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 1 FOLLOWER]: Refusing update from remote peer 7ea6064487bc48d7a050babd80852d04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:27.854549 19067 consensus_queue.cc:1035] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:27.854676 19068 consensus_queue.cc:1035] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:27.858282 19073 mvcc.cc:205] Tried to move back new op lower bound from 7333734104488239104 to 7333734104324136960. Current Snapshot: MvccSnapshot[applied={T|T < 7333734104488239104}]
I20260926 22:40:27.860121 17787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:27.860208 17787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:27.861510 17787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce9c66c085544744a0b3ba6679cd761e: Sending DeleteTablet for 3 replicas of tablet 7d31318e05904afd9f2d485eed9d2991
I20260926 22:40:27.861804 18959 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d31318e05904afd9f2d485eed9d2991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:27 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:38102
I20260926 22:40:27.861914 18687 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d31318e05904afd9f2d485eed9d2991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:27 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:36724
I20260926 22:40:27.862385 18823 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d31318e05904afd9f2d485eed9d2991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:27 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:34592
I20260926 22:40:27.862733 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 17756
I20260926 22:40:27.862890 19081 tablet_replica.cc:330] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225: stopping tablet replica
I20260926 22:40:27.867018 19081 raft_consensus.cc:2229] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:27.867136 19081 raft_consensus.cc:2258] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:27.867415 19082 tablet_replica.cc:330] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb: stopping tablet replica
I20260926 22:40:27.867533 19082 raft_consensus.cc:2229] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:27.867640 19082 raft_consensus.cc:2258] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:27.867961 19081 ts_tablet_manager.cc:1905] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:27.871122 19082 ts_tablet_manager.cc:1905] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:27.862602 19080 tablet_replica.cc:330] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04: stopping tablet replica
I20260926 22:40:27.871243 19080 raft_consensus.cc:2229] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:27.871378 19080 raft_consensus.cc:2258] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:27.871824 19080 ts_tablet_manager.cc:1905] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:27.872997 19082 ts_tablet_manager.cc:1918] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:27.873078 19082 log.cc:1196] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/7d31318e05904afd9f2d485eed9d2991
I20260926 22:40:27.873381 19082 ts_tablet_manager.cc:1939] T 7d31318e05904afd9f2d485eed9d2991 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Deleting consensus metadata
I20260926 22:40:27.873594 19080 ts_tablet_manager.cc:1918] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:27.873665 19080 log.cc:1196] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/7d31318e05904afd9f2d485eed9d2991
W20260926 22:40:27.873771 18779 connection.cc:441] server connection from 127.0.0.1:34592 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34592 (request call id 1) could send its response
I20260926 22:40:27.873979 19080 ts_tablet_manager.cc:1939] T 7d31318e05904afd9f2d485eed9d2991 P 7ea6064487bc48d7a050babd80852d04: Deleting consensus metadata
I20260926 22:40:27.874333 19081 ts_tablet_manager.cc:1918] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 22:40:27.874348 18912 connection.cc:441] server connection from 127.0.0.1:38102 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38102 (request call id 1) could send its response
I20260926 22:40:27.874415 19081 log.cc:1196] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-0/wal/wals/7d31318e05904afd9f2d485eed9d2991
I20260926 22:40:27.874708 19081 ts_tablet_manager.cc:1939] T 7d31318e05904afd9f2d485eed9d2991 P 5c80eb7981614c99afa3149d5f4cc225: Deleting consensus metadata
W20260926 22:40:27.875108 18641 connection.cc:441] server connection from 127.0.0.1:36724 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36724 (request call id 1) could send its response
I20260926 22:40:27.882817 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.874614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.11.107.253:36977)
0926 22:40:27.874661 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.874663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.874810 (+ 147us) server_negotiation.cc:407] Connection header received
0926 22:40:27.874921 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.874922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.874935 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.875011 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.875927 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.876718 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.877585 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.877804 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.877917 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.879595 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.879596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.879596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.879603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.881485 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.881917 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.881918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.881919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.881981 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.882209 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.882210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.882210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.882362 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.882464 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.882619 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.882717 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:40:27.883141 19036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:27.875652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.11.107.252:35439)
0926 22:40:27.875697 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:27.875699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:27.875751 (+ 52us) server_negotiation.cc:407] Connection header received
0926 22:40:27.875847 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:27.875848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:27.875863 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:27.875896 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:27.876494 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.877148 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.878253 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:27.878562 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:27.878618 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:27.880142 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:27.880143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:27.880144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:27.880152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:27.881822 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.882270 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.882271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.882272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.882323 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:27.882537 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:27.882538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:27.882538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:27.882673 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:27.882764 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:27.882959 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 22:40:27.883056 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:40:28.858407 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:28.858016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.195:54157)
0926 22:40:28.858229 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:28.858344 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":60,"threads_started":1}
W20260926 22:40:28.858713 19026 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:32867 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
I20260926 22:40:28.858716 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:28.858364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.193:36811)
0926 22:40:28.858604 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:28.858671 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":47,"threads_started":1}
W20260926 22:40:28.858989 18754 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:32867 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
I20260926 22:40:28.859239 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:28.859023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.194:39573)
0926 22:40:28.859129 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:28.859198 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":50,"threads_started":1}
W20260926 22:40:28.859447 18890 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:32867 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
I20260926 22:40:29.362087 19092 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce9c66c085544744a0b3ba6679cd761e)
I20260926 22:40:29.362316 19093 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce9c66c085544744a0b3ba6679cd761e)
I20260926 22:40:29.362337 19092 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } }
I20260926 22:40:29.362429 19093 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } }
I20260926 22:40:29.362671 19093 leader_election.cc:288] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce9c66c085544744a0b3ba6679cd761e (127.11.107.254:32867), b7a1144270434ce398847d0d7b7c144b (127.11.107.252:35439)
I20260926 22:40:29.362980 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.362795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:38192)
0926 22:40:29.362897 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.362939 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
I20260926 22:40:29.362991 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.362774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:38202)
0926 22:40:29.362886 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.362940 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":47,"threads_started":1}
W20260926 22:40:29.363112 18370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
W20260926 22:40:29.363377 18556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
I20260926 22:40:29.363595 19092 leader_election.cc:288] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce9c66c085544744a0b3ba6679cd761e (127.11.107.254:32867), ab6c05c274ba4e3989741959ee16ed82 (127.11.107.253:36977)
I20260926 22:40:29.363658 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.363504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:38210)
0926 22:40:29.363608 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.363631 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
W20260926 22:40:29.363751 18370 leader_election.cc:334] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ce9c66c085544744a0b3ba6679cd761e (127.11.107.254:32867): Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
I20260926 22:40:29.364171 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.363971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:38212)
0926 22:40:29.364106 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.364141 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
W20260926 22:40:29.364243 18556 leader_election.cc:334] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ce9c66c085544744a0b3ba6679cd761e (127.11.107.254:32867): Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
I20260926 22:40:29.365954 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.363104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.11.107.252:35439)
0926 22:40:29.363310 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:29.363312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:29.363318 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:40:29.363342 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:29.363343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:29.363353 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:29.363408 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:29.363600 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.364346 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.365280 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.365641 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.365700 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:29.365708 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:29.365855 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 22:40:29.365901 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":41,"threads_started":1}
I20260926 22:40:29.365952 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.362939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:35439 (local address 127.0.0.1:57842)
0926 22:40:29.363122 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.363132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:29.363207 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:29.363458 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:29.363460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:29.363462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:29.363521 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:29.363523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.364428 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.364429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:29.365193 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:29.365195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.365714 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.365715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:29.365773 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:29.365776 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:29.365827 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:40:29.365886 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":35,"threads_started":1}
I20260926 22:40:29.366091 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.363080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:36977 (local address 127.0.0.1:36334)
0926 22:40:29.363238 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.363248 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:29.363299 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:29.363606 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:29.363607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:29.363609 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:29.363661 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:29.363663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.364509 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.364509 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:29.365223 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:29.365225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.365814 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.365814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:29.365911 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:29.365915 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:29.365966 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:40:29.366010 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260926 22:40:29.366215 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.363246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36334 (local address 127.11.107.253:36977)
0926 22:40:29.363471 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:29.363473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:29.363482 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:29.363503 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:29.363505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:29.363513 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:29.363723 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:29.363757 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.364434 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.365337 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.365697 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.365786 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:29.365794 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:29.366080 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 22:40:29.366123 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
I20260926 22:40:29.366288 18582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab6c05c274ba4e3989741959ee16ed82" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b7a1144270434ce398847d0d7b7c144b" is_pre_election: true
I20260926 22:40:29.366384 18582 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6c05c274ba4e3989741959ee16ed82 in term 1.
I20260926 22:40:29.366552 18371 leader_election.cc:302] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab6c05c274ba4e3989741959ee16ed82, b7a1144270434ce398847d0d7b7c144b; no voters: ce9c66c085544744a0b3ba6679cd761e
I20260926 22:40:29.366698 19093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 22:40:29.366674 18396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7a1144270434ce398847d0d7b7c144b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ab6c05c274ba4e3989741959ee16ed82" is_pre_election: true
I20260926 22:40:29.366732 19093 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce9c66c085544744a0b3ba6679cd761e)
I20260926 22:40:29.366760 19093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:40:29.367528 19093 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } }
I20260926 22:40:29.367729 18396 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b7a1144270434ce398847d0d7b7c144b in current term 2: Already voted for candidate ab6c05c274ba4e3989741959ee16ed82 in this term.
I20260926 22:40:29.367760 19093 leader_election.cc:288] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [CANDIDATE]: Term 2 election: Requested vote from peers ce9c66c085544744a0b3ba6679cd761e (127.11.107.254:32867), b7a1144270434ce398847d0d7b7c144b (127.11.107.252:35439)
I20260926 22:40:29.367949 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.367830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:38228)
0926 22:40:29.367886 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.367921 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 22:40:29.368093 18582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab6c05c274ba4e3989741959ee16ed82" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b7a1144270434ce398847d0d7b7c144b"
I20260926 22:40:29.368160 18559 leader_election.cc:302] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7a1144270434ce398847d0d7b7c144b; no voters: ab6c05c274ba4e3989741959ee16ed82, ce9c66c085544744a0b3ba6679cd761e
I20260926 22:40:29.368165 18582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:40:29.368296 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.368205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:38230)
0926 22:40:29.368250 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.368273 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 22:40:29.368347 18370 leader_election.cc:334] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ce9c66c085544744a0b3ba6679cd761e (127.11.107.254:32867): Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
I20260926 22:40:29.368944 18582 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6c05c274ba4e3989741959ee16ed82 in term 2.
I20260926 22:40:29.369052 19092 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:40:29.369205 18371 leader_election.cc:302] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab6c05c274ba4e3989741959ee16ed82, b7a1144270434ce398847d0d7b7c144b; no voters: ce9c66c085544744a0b3ba6679cd761e
I20260926 22:40:29.369292 19093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 22:40:29.369423 19093 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [term 2 LEADER]: Becoming Leader. State: Replica: ab6c05c274ba4e3989741959ee16ed82, State: Running, Role: LEADER
I20260926 22:40:29.369513 19093 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } }
I20260926 22:40:29.369827 19103 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab6c05c274ba4e3989741959ee16ed82. Latest consensus state: current_term: 2 leader_uuid: "ab6c05c274ba4e3989741959ee16ed82" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } } }
I20260926 22:40:29.369949 19103 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:29.370088 19105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:29.370486 19105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:29.370576 19105 catalog_manager.cc:1259] Loaded cluster ID: c4f1ae9664b646ac9c393356187e9503
I20260926 22:40:29.370607 19105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:29.370788 19105 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:29.370883 19105 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: Loaded TSK: 0
I20260926 22:40:29.371021 19105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:40:29.425664 18386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:29.427322 18386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:29.428406 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.428277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:38242)
0926 22:40:29.428333 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.428370 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 22:40:29.428413 18582 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [term 2 FOLLOWER]: Refusing update from remote peer ab6c05c274ba4e3989741959ee16ed82: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 22:40:29.428772 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.428672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.0.0.1:38250)
0926 22:40:29.428729 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.428754 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 22:40:29.428735 19093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 22:40:29.428850 18370 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 -> Peer ce9c66c085544744a0b3ba6679cd761e (127.11.107.254:32867): Couldn't send request to peer ce9c66c085544744a0b3ba6679cd761e. Status: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 22:40:29.429394 19092 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab6c05c274ba4e3989741959ee16ed82. Latest consensus state: current_term: 2 leader_uuid: "ab6c05c274ba4e3989741959ee16ed82" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } } }
I20260926 22:40:29.429497 19092 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:40:29.430063 19107 mvcc.cc:205] Tried to move back new op lower bound from 7333734110936846336 to 7333734110697959424. Current Snapshot: MvccSnapshot[applied={T|T < 7333734110936846336}]
I20260926 22:40:29.430083 19093 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ab6c05c274ba4e3989741959ee16ed82" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } } }
I20260926 22:40:29.430166 19093 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:29.430287 19092 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ab6c05c274ba4e3989741959ee16ed82" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce9c66c085544744a0b3ba6679cd761e" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 32867 } } peers { permanent_uuid: "ab6c05c274ba4e3989741959ee16ed82" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 36977 } attrs { promote: false } } peers { permanent_uuid: "b7a1144270434ce398847d0d7b7c144b" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 35439 } attrs { promote: false } } }
I20260926 22:40:29.430378 19092 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a1144270434ce398847d0d7b7c144b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:40:29.435105 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.431810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:33873 (local address 127.0.0.1:38134)
0926 22:40:29.431942 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.431951 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:29.431998 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:29.432154 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:29.432156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:29.432157 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:29.432211 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:29.432213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.432950 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.432951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:29.434033 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:29.434035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.434877 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.434878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:29.434943 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:29.434947 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:29.434977 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 22:40:29.435023 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 22:40:29.435261 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.431959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.11.107.193:44109)
0926 22:40:29.432067 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:29.432069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:29.432114 (+ 45us) server_negotiation.cc:407] Connection header received
0926 22:40:29.432178 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:29.432179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:29.432188 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:29.432567 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:29.432603 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.433319 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.434351 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.434888 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.434983 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:29.434993 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:29.435151 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 22:40:29.435194 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260926 22:40:29.435261 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.431764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:44109 (local address 127.0.0.1:36740)
0926 22:40:29.432059 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.432069 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:29.432124 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:29.432475 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:29.432475 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:29.432477 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:29.432510 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:29.432510 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.433383 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.433384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:29.434258 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:29.434261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.434984 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.434985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:29.435049 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:29.435053 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:29.435137 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 22:40:29.435181 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":137}
I20260926 22:40:29.435527 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.431911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.11.107.195:33873)
0926 22:40:29.432018 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:29.432021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:29.432032 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:40:29.432060 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:29.432061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:29.432074 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:29.432141 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:29.432294 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.432866 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.434174 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.434782 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.435314 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:29.435369 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:29.435389 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 22:40:29.435448 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":54,"threads_started":1}
I20260926 22:40:29.435540 18687 tablet_service.cc:1467] Processing CreateTablet for tablet 98e29c7874614557b2a66d5c88c9e9e3 (DEFAULT_TABLE table=test-table [id=3547e4b199d34c12a434dd846500ba7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:29.435708 18687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e29c7874614557b2a66d5c88c9e9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:29.435720 18959 tablet_service.cc:1467] Processing CreateTablet for tablet 98e29c7874614557b2a66d5c88c9e9e3 (DEFAULT_TABLE table=test-table [id=3547e4b199d34c12a434dd846500ba7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:29.435813 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.431764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35919 (local address 127.0.0.1:34600)
0926 22:40:29.431808 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.431818 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:29.431854 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:29.432345 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:29.432346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:29.432347 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:29.432378 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:29.432380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.433308 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.433309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:29.434388 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:29.434390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.435594 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.435594 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:29.435661 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:29.435666 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:40:29.435703 (+ 37us) client_negotiation.cc:240] Negotiation successful
0926 22:40:29.435748 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 22:40:29.435835 18959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e29c7874614557b2a66d5c88c9e9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:29.436139 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.431859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.11.107.194:35919)
0926 22:40:29.432218 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:29.432220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:29.432227 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:29.432266 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:29.432267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:29.432278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:29.432344 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:29.432462 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.433228 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.434474 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.435087 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.435995 (+ 908us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:29.436006 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:29.436025 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 22:40:29.436069 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":58,"threads_started":1}
I20260926 22:40:29.436344 18823 tablet_service.cc:1467] Processing CreateTablet for tablet 98e29c7874614557b2a66d5c88c9e9e3 (DEFAULT_TABLE table=test-table [id=3547e4b199d34c12a434dd846500ba7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:29.436537 18823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e29c7874614557b2a66d5c88c9e9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:29.437314 19115 tablet_bootstrap.cc:492] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225: Bootstrap starting.
I20260926 22:40:29.437674 19115 tablet_bootstrap.cc:654] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:29.437960 19116 tablet_bootstrap.cc:492] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04: Bootstrap starting.
I20260926 22:40:29.438176 19118 tablet_bootstrap.cc:492] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Bootstrap starting.
I20260926 22:40:29.438256 19115 tablet_bootstrap.cc:492] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225: No bootstrap required, opened a new log
I20260926 22:40:29.438300 19115 ts_tablet_manager.cc:1397] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:29.438360 19116 tablet_bootstrap.cc:654] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:29.438436 19115 raft_consensus.cc:348] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } }
I20260926 22:40:29.438521 19115 raft_consensus.cc:374] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:29.438546 19115 raft_consensus.cc:729] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c80eb7981614c99afa3149d5f4cc225, State: Initialized, Role: FOLLOWER
I20260926 22:40:29.438575 19118 tablet_bootstrap.cc:654] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:29.438601 19115 consensus_queue.cc:260] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } }
I20260926 22:40:29.438794 19115 ts_tablet_manager.cc:1428] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:29.439073 19116 tablet_bootstrap.cc:492] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04: No bootstrap required, opened a new log
I20260926 22:40:29.439121 19116 ts_tablet_manager.cc:1397] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:29.439229 19116 raft_consensus.cc:348] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } }
I20260926 22:40:29.439291 19116 raft_consensus.cc:374] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:29.439317 19116 raft_consensus.cc:729] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ea6064487bc48d7a050babd80852d04, State: Initialized, Role: FOLLOWER
I20260926 22:40:29.439363 19116 consensus_queue.cc:260] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } }
I20260926 22:40:29.439482 19116 ts_tablet_manager.cc:1428] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:29.439898 18755 heartbeater.cc:503] Master 127.11.107.253:36977 was elected leader, sending a full tablet report...
I20260926 22:40:29.440007 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.439730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.195:56109)
0926 22:40:29.439928 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.439970 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260926 22:40:29.440217 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.439050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.193:37097)
0926 22:40:29.439820 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.440155 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":68,"threads_started":1}
I20260926 22:40:29.440469 19118 tablet_bootstrap.cc:492] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb: No bootstrap required, opened a new log
I20260926 22:40:29.440527 19118 ts_tablet_manager.cc:1397] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:29.440619 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.440535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.195:39143)
0926 22:40:29.440571 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.440597 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 22:40:29.440676 19118 raft_consensus.cc:348] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } }
I20260926 22:40:29.440755 19118 raft_consensus.cc:374] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:29.440776 19118 raft_consensus.cc:729] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ce7f00f3424cf0982c2b4de5e5cdeb, State: Initialized, Role: FOLLOWER
I20260926 22:40:29.440783 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.440478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.193:33237)
0926 22:40:29.440731 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.440759 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237}
I20260926 22:40:29.440827 19118 consensus_queue.cc:260] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } }
I20260926 22:40:29.440896 19027 heartbeater.cc:503] Master 127.11.107.253:36977 was elected leader, sending a full tablet report...
I20260926 22:40:29.441002 19118 ts_tablet_manager.cc:1428] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:29.441458 18891 heartbeater.cc:503] Master 127.11.107.253:36977 was elected leader, sending a full tablet report...
I20260926 22:40:29.441520 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.441200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.194:41927)
0926 22:40:29.441413 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.441451 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
I20260926 22:40:29.441908 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.441769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.194:50451)
0926 22:40:29.441806 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.441879 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 22:40:29.630234 18756 tablet.cc:2369] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:40:29.644630 19028 tablet.cc:2369] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:40:29.735157 19123 raft_consensus.cc:482] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:29.735240 19123 raft_consensus.cc:504] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } }
I20260926 22:40:29.735417 19123 leader_election.cc:288] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c80eb7981614c99afa3149d5f4cc225 (127.11.107.193:44109), b0ce7f00f3424cf0982c2b4de5e5cdeb (127.11.107.194:35919)
I20260926 22:40:29.735675 18707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e29c7874614557b2a66d5c88c9e9e3" candidate_uuid: "7ea6064487bc48d7a050babd80852d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c80eb7981614c99afa3149d5f4cc225" is_pre_election: true
I20260926 22:40:29.735666 18843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e29c7874614557b2a66d5c88c9e9e3" candidate_uuid: "7ea6064487bc48d7a050babd80852d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" is_pre_election: true
I20260926 22:40:29.735764 18843 raft_consensus.cc:2454] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ea6064487bc48d7a050babd80852d04 in term 0.
I20260926 22:40:29.735764 18707 raft_consensus.cc:2454] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ea6064487bc48d7a050babd80852d04 in term 0.
I20260926 22:40:29.735946 18912 leader_election.cc:302] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c80eb7981614c99afa3149d5f4cc225, 7ea6064487bc48d7a050babd80852d04; no voters:
I20260926 22:40:29.736058 19123 raft_consensus.cc:2781] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:29.736099 19123 raft_consensus.cc:482] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:29.736148 19123 raft_consensus.cc:3037] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:29.736968 19123 raft_consensus.cc:504] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } }
I20260926 22:40:29.737106 19123 leader_election.cc:288] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [CANDIDATE]: Term 1 election: Requested vote from peers 5c80eb7981614c99afa3149d5f4cc225 (127.11.107.193:44109), b0ce7f00f3424cf0982c2b4de5e5cdeb (127.11.107.194:35919)
I20260926 22:40:29.737242 18707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e29c7874614557b2a66d5c88c9e9e3" candidate_uuid: "7ea6064487bc48d7a050babd80852d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c80eb7981614c99afa3149d5f4cc225"
I20260926 22:40:29.737264 18843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e29c7874614557b2a66d5c88c9e9e3" candidate_uuid: "7ea6064487bc48d7a050babd80852d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb"
I20260926 22:40:29.737329 18707 raft_consensus.cc:3037] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:29.737329 18843 raft_consensus.cc:3037] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:29.738094 18707 raft_consensus.cc:2454] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ea6064487bc48d7a050babd80852d04 in term 1.
I20260926 22:40:29.738094 18843 raft_consensus.cc:2454] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ea6064487bc48d7a050babd80852d04 in term 1.
I20260926 22:40:29.738258 18913 leader_election.cc:302] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ea6064487bc48d7a050babd80852d04, b0ce7f00f3424cf0982c2b4de5e5cdeb; no voters:
I20260926 22:40:29.738346 19123 raft_consensus.cc:2781] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:29.738517 19123 raft_consensus.cc:686] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [term 1 LEADER]: Becoming Leader. State: Replica: 7ea6064487bc48d7a050babd80852d04, State: Running, Role: LEADER
I20260926 22:40:29.738569 19123 consensus_queue.cc:237] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } }
I20260926 22:40:29.738883 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.738762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.195:53307)
0926 22:40:29.738804 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.738847 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 22:40:29.739374 18385 catalog_manager.cc:5495] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ea6064487bc48d7a050babd80852d04 (127.11.107.195). New cstate: current_term: 1 leader_uuid: "7ea6064487bc48d7a050babd80852d04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ea6064487bc48d7a050babd80852d04" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33873 } health_report { overall_health: HEALTHY } } }
I20260926 22:40:29.786818 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.784661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.11.107.195:33873)
0926 22:40:29.784723 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:29.784725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:29.784741 (+ 16us) server_negotiation.cc:407] Connection header received
0926 22:40:29.784763 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:29.784763 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:29.784772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:29.784820 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:29.784943 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.785423 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.786224 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:29.786418 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:29.786482 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:29.786610 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:29.786637 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:29.786678 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:29.786740 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 22:40:29.786760 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 22:40:29.787683 18843 raft_consensus.cc:1261] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 1 FOLLOWER]: Refusing update from remote peer 7ea6064487bc48d7a050babd80852d04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:29.787710 18707 raft_consensus.cc:1261] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225 [term 1 FOLLOWER]: Refusing update from remote peer 7ea6064487bc48d7a050babd80852d04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:29.787909 19127 consensus_queue.cc:1035] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0ce7f00f3424cf0982c2b4de5e5cdeb" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:29.788066 19123 consensus_queue.cc:1035] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c80eb7981614c99afa3149d5f4cc225" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:29.788547 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.788443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.194:58703)
0926 22:40:29.788489 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.788518 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 22:40:29.789434 19130 mvcc.cc:205] Tried to move back new op lower bound from 7333734112408813568 to 7333734112209879040. Current Snapshot: MvccSnapshot[applied={T|T < 7333734112408813568}]
I20260926 22:40:29.789547 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.789446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.195:49047)
0926 22:40:29.789486 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.789514 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 22:40:29.789619 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.789489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.193:47027)
0926 22:40:29.789559 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.789588 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 22:40:29.790294 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.790047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.194:41469)
0926 22:40:29.790241 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.790269 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260926 22:40:29.791391 19133 mvcc.cc:205] Tried to move back new op lower bound from 7333734112408813568 to 7333734112209879040. Current Snapshot: MvccSnapshot[applied={T|T < 7333734112408813568}]
I20260926 22:40:29.791503 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.791413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.193:44497)
0926 22:40:29.791450 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.791473 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 22:40:29.791834 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.791724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:32867 (local address 127.11.107.195:47247)
0926 22:40:29.791777 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.791808 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 22:40:29.792474 18386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:29.792546 18386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:29.793704 18386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab6c05c274ba4e3989741959ee16ed82: Sending DeleteTablet for 3 replicas of tablet 98e29c7874614557b2a66d5c88c9e9e3
I20260926 22:40:29.794023 18823 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e29c7874614557b2a66d5c88c9e9e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:29 UTC) from {username='oryx', principal='oryx/127.11.107.253@KRBTEST.COM'} at 127.0.0.1:34600
I20260926 22:40:29.794094 18687 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e29c7874614557b2a66d5c88c9e9e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:29 UTC) from {username='oryx', principal='oryx/127.11.107.253@KRBTEST.COM'} at 127.0.0.1:36740
I20260926 22:40:29.794170 18959 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e29c7874614557b2a66d5c88c9e9e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:29 UTC) from {username='oryx', principal='oryx/127.11.107.253@KRBTEST.COM'} at 127.0.0.1:38134
I20260926 22:40:29.794271 19140 tablet_replica.cc:330] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225: stopping tablet replica
I20260926 22:40:29.794322 19140 raft_consensus.cc:2229] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:29.794358 19140 raft_consensus.cc:2258] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:29.794358 19139 tablet_replica.cc:330] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb: stopping tablet replica
I20260926 22:40:29.794404 19139 raft_consensus.cc:2229] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:29.794446 19139 raft_consensus.cc:2258] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:29.794631 19140 ts_tablet_manager.cc:1905] T 98e29c7874614557b2a66d5c88c9e9e3 P 5c80eb7981614c99afa3149d5f4cc225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:29.794700 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 18624
I20260926 22:40:29.794721 19139 ts_tablet_manager.cc:1905] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:29.797003 19139 ts_tablet_manager.cc:1918] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:29.797058 19139 log.cc:1196] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/98e29c7874614557b2a66d5c88c9e9e3
I20260926 22:40:29.797282 19139 ts_tablet_manager.cc:1939] T 98e29c7874614557b2a66d5c88c9e9e3 P b0ce7f00f3424cf0982c2b4de5e5cdeb: Deleting consensus metadata
I20260926 22:40:29.797582 18371 catalog_manager.cc:4841] TS b0ce7f00f3424cf0982c2b4de5e5cdeb (127.11.107.194:35919): tablet 98e29c7874614557b2a66d5c88c9e9e3 (table test-table [id=3547e4b199d34c12a434dd846500ba7c]) successfully deleted
I20260926 22:40:29.801100 19141 tablet_replica.cc:330] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04: stopping tablet replica
I20260926 22:40:29.801177 19141 raft_consensus.cc:2229] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:29.801237 19141 raft_consensus.cc:2258] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:29.801502 19141 ts_tablet_manager.cc:1905] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:29.801941 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:29.801832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:44109 (local address 127.0.0.1:36750)
0926 22:40:29.801890 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:29.801915 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:44109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 22:40:29.802065 18370 catalog_manager.cc:4584] TS 5c80eb7981614c99afa3149d5f4cc225 (127.11.107.193:44109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98e29c7874614557b2a66d5c88c9e9e3: Network error: Client connection negotiation failed: client connection to 127.11.107.193:44109: connect: Connection refused (error 111)
I20260926 22:40:29.802102 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 18760
I20260926 22:40:29.803267 19141 ts_tablet_manager.cc:1918] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:29.803329 19141 log.cc:1196] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/98e29c7874614557b2a66d5c88c9e9e3
I20260926 22:40:29.803563 19141 ts_tablet_manager.cc:1939] T 98e29c7874614557b2a66d5c88c9e9e3 P 7ea6064487bc48d7a050babd80852d04: Deleting consensus metadata
I20260926 22:40:29.803864 18372 catalog_manager.cc:4841] TS 7ea6064487bc48d7a050babd80852d04 (127.11.107.195:33873): tablet 98e29c7874614557b2a66d5c88c9e9e3 (table test-table [id=3547e4b199d34c12a434dd846500ba7c]) successfully deleted
I20260926 22:40:29.806902 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 18896
I20260926 22:40:29.813131 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 18252
I20260926 22:40:29.817871 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 18440
2026-09-26T22:40:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5003 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:29 dist-test-slave-j8sq krb5kdc[19144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:29 dist-test-slave-j8sq krb5kdc[19144](info): set up 2 sockets
Sep 26 22:40:29 dist-test-slave-j8sq krb5kdc[19144](info): commencing operation
krb5kdc: starting...
W20260926 22:40:30.847255 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:30 dist-test-slave-j8sq krb5kdc[19144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.254.keytab.
Entry for principal oryx/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.254.keytab.
I20260926 22:40:30.965531 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:35651
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.254.keytab
--principal=oryx/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:35651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:31.045789 19157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:31.046027 19157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:31.046057 19157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:31.047158 19157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:31.047190 19157 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 22:40:31.047204 19157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:31.047216 19157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:31.047228 19157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:31.047240 19157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:31.048707 19157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:35651
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.254.keytab
--principal=oryx/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:35651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:31.048898 19157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:31.049070 19157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:40:31.053831 19157 server_base.cc:1034] running on GCE node
W20260926 22:40:31.053802 19165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:31.053797 19163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:31.054064 19162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:31.054387 19157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:31.054677 19157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:31.055864 19157 hybrid_clock.cc:648] HybridClock initialized: now 1790462431055841 us; error 42 us; skew 500 ppm
Sep 26 22:40:31 dist-test-slave-j8sq krb5kdc[19144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462431, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:31.059573 19157 init.cc:377] Logged in from keytab as oryx/127.11.107.254@KRBTEST.COM (short username oryx)
I20260926 22:40:31.060273 19157 webserver.cc:460] Webserver started at http://127.11.107.254:42499/ using document root <none> and password file <none>
I20260926 22:40:31.060648 19157 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:31.060721 19157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:31.060941 19157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:31.062503 19157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "951459dc250d4853b29b3cd022ac5287"
format_stamp: "Formatted at 2026-09-26 22:40:31 on dist-test-slave-j8sq"
server_key: "379969f1de6a0979a5cd1277e044fdd4"
server_key_iv: "09e2ab2e37fa026d7f70ab7d05abbafe"
server_key_version: "encryptionkey@0"
I20260926 22:40:31.062912 19157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "951459dc250d4853b29b3cd022ac5287"
format_stamp: "Formatted at 2026-09-26 22:40:31 on dist-test-slave-j8sq"
server_key: "379969f1de6a0979a5cd1277e044fdd4"
server_key_iv: "09e2ab2e37fa026d7f70ab7d05abbafe"
server_key_version: "encryptionkey@0"
I20260926 22:40:31.064396 19157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:31.065119 19172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:31.065285 19157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:31.065387 19157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "951459dc250d4853b29b3cd022ac5287"
format_stamp: "Formatted at 2026-09-26 22:40:31 on dist-test-slave-j8sq"
server_key: "379969f1de6a0979a5cd1277e044fdd4"
server_key_iv: "09e2ab2e37fa026d7f70ab7d05abbafe"
server_key_version: "encryptionkey@0"
I20260926 22:40:31.065469 19157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:31.076658 19157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:31.078522 19157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:31.078693 19157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:31.083303 19157 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:35651
I20260926 22:40:31.083384 19234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:35651 every 8 connection(s)
I20260926 22:40:31.083727 19157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:31.084442 19235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:31.086771 19235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287: Bootstrap starting.
I20260926 22:40:31.087311 19235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:31.087548 19235 log.cc:824] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:31.088158 19235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287: No bootstrap required, opened a new log
I20260926 22:40:31.090198 19235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951459dc250d4853b29b3cd022ac5287" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35651 } }
I20260926 22:40:31.090348 19235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:31.090322 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 19157
I20260926 22:40:31.090371 19235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951459dc250d4853b29b3cd022ac5287, State: Initialized, Role: FOLLOWER
I20260926 22:40:31.090433 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:31.090534 19235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951459dc250d4853b29b3cd022ac5287" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35651 } }
I20260926 22:40:31.090605 19235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:31.090662 19235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:31.090673 11695 external_mini_cluster.cc:1428] Setting key 1db343dbf44023538fe7385dca6ed7fe
I20260926 22:40:31.090709 19235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:31.091349 19235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951459dc250d4853b29b3cd022ac5287" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35651 } }
I20260926 22:40:31.091459 19235 leader_election.cc:302] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 951459dc250d4853b29b3cd022ac5287; no voters:
I20260926 22:40:31.091662 19235 leader_election.cc:288] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:31.091687 19240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:31.091792 19240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [term 1 LEADER]: Becoming Leader. State: Replica: 951459dc250d4853b29b3cd022ac5287, State: Running, Role: LEADER
I20260926 22:40:31.091867 19240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951459dc250d4853b29b3cd022ac5287" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35651 } }
I20260926 22:40:31.091918 19235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:31.092157 19240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "951459dc250d4853b29b3cd022ac5287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951459dc250d4853b29b3cd022ac5287" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35651 } } }
I20260926 22:40:31.092214 19240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:31.092340 19240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 951459dc250d4853b29b3cd022ac5287. Latest consensus state: current_term: 1 leader_uuid: "951459dc250d4853b29b3cd022ac5287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951459dc250d4853b29b3cd022ac5287" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35651 } } }
I20260926 22:40:31.092397 19240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:31.092806 19244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:31.093452 19244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:31.095959 19244 catalog_manager.cc:1347] Generated new cluster ID: 6d1a3e45f5ad48e2b02affa922b330da
I20260926 22:40:31.096010 19244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:40:31 dist-test-slave-j8sq krb5kdc[19144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
I20260926 22:40:31.102598 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.091205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.11.107.254:35651)
0926 22:40:31.091377 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.091385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.091410 (+ 25us) server_negotiation.cc:407] Connection header received
0926 22:40:31.091537 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.091542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.091610 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.091781 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:31.092557 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.093984 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.094832 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.095440 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.095560 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.098425 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:31.098429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:31.098430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:31.098442 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:31.101005 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.101378 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.101382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.101382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.101492 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.101691 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.101692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.101692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.102018 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:31.102179 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.102256 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.102390 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260926 22:40:31.134891 19244 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:31.135456 19244 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:31.140725 19244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287: Generated new TSK 0
I20260926 22:40:31.140926 19244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab.
Entry for principal oryx/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab.
I20260926 22:40:31.172477 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:35651
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab
--principal=oryx/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:31.252650 19261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:31.252838 19261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:31.252861 19261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:31.253966 19261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:31.254001 19261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:31.254020 19261 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:40:31.254037 19261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:31.255432 19261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab
--principal=oryx/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.107.254:35651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:31.255605 19261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:31.255775 19261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:31.260561 19266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:31.260556 19269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:31.260556 19267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:31.260994 19261 server_base.cc:1034] running on GCE node
I20260926 22:40:31.261200 19261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:31.261380 19261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:31.262516 19261 hybrid_clock.cc:648] HybridClock initialized: now 1790462431262505 us; error 26 us; skew 500 ppm
Sep 26 22:40:31 dist-test-slave-j8sq krb5kdc[19144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462431, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:31.265002 19261 init.cc:377] Logged in from keytab as oryx/127.11.107.193@KRBTEST.COM (short username oryx)
I20260926 22:40:31.265440 19261 webserver.cc:460] Webserver started at http://127.11.107.193:40977/ using document root <none> and password file <none>
I20260926 22:40:31.265614 19261 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:31.265651 19261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:31.265738 19261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:31.266836 19261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "e8f02ff9af9642578f2289ddbdd0f08b"
format_stamp: "Formatted at 2026-09-26 22:40:31 on dist-test-slave-j8sq"
server_key: "a89b9a0e6b494dad243f0d0d08de3241"
server_key_iv: "407ca0882581d8be09324f81e8143a2e"
server_key_version: "encryptionkey@0"
I20260926 22:40:31.267130 19261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "e8f02ff9af9642578f2289ddbdd0f08b"
format_stamp: "Formatted at 2026-09-26 22:40:31 on dist-test-slave-j8sq"
server_key: "a89b9a0e6b494dad243f0d0d08de3241"
server_key_iv: "407ca0882581d8be09324f81e8143a2e"
server_key_version: "encryptionkey@0"
I20260926 22:40:31.268457 19261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:31.269150 19276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:31.269339 19261 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:31.269428 19261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "e8f02ff9af9642578f2289ddbdd0f08b"
format_stamp: "Formatted at 2026-09-26 22:40:31 on dist-test-slave-j8sq"
server_key: "a89b9a0e6b494dad243f0d0d08de3241"
server_key_iv: "407ca0882581d8be09324f81e8143a2e"
server_key_version: "encryptionkey@0"
I20260926 22:40:31.269503 19261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:31.291908 19261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:31.294428 19261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:31.294576 19261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:31.295143 19261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:31.295190 19261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:31.295213 19261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:31.295226 19261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 22:40:31 dist-test-slave-j8sq krb5kdc[19144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462431, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
I20260926 22:40:31.302261 19261 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:40265
I20260926 22:40:31.302634 19261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:31.302656 19391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:40265 every 8 connection(s)
I20260926 22:40:31.304427 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.295751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:48915 (local address 127.11.107.254:35651)
0926 22:40:31.295817 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.295819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.295834 (+ 15us) server_negotiation.cc:407] Connection header received
0926 22:40:31.296365 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.296367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.296382 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.296433 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:31.297197 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.297992 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.298993 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.299245 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.299326 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.301802 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:31.301803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:31.301804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:31.301813 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:31.303340 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.303701 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.303702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.303702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.303753 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.303925 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.303925 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.303926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.304019 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:31.304068 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.304305 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.304365 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 22:40:31.304605 19285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.295573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:35651 (local address 127.11.107.193:48915)
0926 22:40:31.295722 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:31.295764 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:31.296251 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:31.296508 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:31.296512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:31.296903 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:31.297100 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.297106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.298079 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.298080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.298895 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.298897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.299335 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.299337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.299457 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.299931 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:31.299939 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:31.301692 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:31.303413 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:31.303417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:31.303418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:31.303628 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:31.303795 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:31.303795 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:31.303796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:31.303876 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:31.304076 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:31.304080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:31.304196 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 22:40:31.304277 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 22:40:31.304388 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260926 22:40:31.305117 19392 heartbeater.cc:347] Connected to a master server at 127.11.107.254:35651
I20260926 22:40:31.305222 19392 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:31.305387 19392 heartbeater.cc:511] Master 127.11.107.254:35651 requested a full tablet report, sending...
I20260926 22:40:31.305974 19189 ts_manager.cc:194] Registered new tserver with Master: e8f02ff9af9642578f2289ddbdd0f08b (127.11.107.193:40265)
I20260926 22:40:31.306599 19189 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:48915
I20260926 22:40:31.307096 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 19261
I20260926 22:40:31.307199 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:31.307435 11695 external_mini_cluster.cc:1428] Setting key 82b1b024416367870e15272722f4186b
I20260926 22:40:31.311518 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.306103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:35651 (local address 127.11.107.193:40913)
0926 22:40:31.306239 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:31.306249 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:31.306318 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:31.306440 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:31.306440 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:31.306597 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:31.306645 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.306647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.307438 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.307439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.308024 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.308026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.308386 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.308387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.308446 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.308853 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:31.308857 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:31.309095 (+ 238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:31.310657 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:31.310657 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:31.310658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:31.310825 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:31.311013 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:31.311014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:31.311014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:31.311061 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:31.311309 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:31.311309 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:31.311342 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 22:40:31.311393 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:40:31.311459 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260926 22:40:31.311625 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.306148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:40913 (local address 127.11.107.254:35651)
0926 22:40:31.306189 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.306191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.306307 (+ 116us) server_negotiation.cc:407] Connection header received
0926 22:40:31.306373 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.306374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.306384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.306436 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:31.306714 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.307332 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.308104 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.308304 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.308385 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.309204 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:31.309205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:31.309206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:31.309211 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:31.310579 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.310893 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.310894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.310894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.310951 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.311118 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.311119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.311119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.311249 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:31.311314 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.311418 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.311499 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for oryx/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab.
Entry for principal oryx/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab.
I20260926 22:40:31.330045 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:35651
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab
--principal=oryx/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:31.409363 19401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:31.409561 19401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:31.409586 19401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:31.410797 19401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:31.410840 19401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:31.410861 19401 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:40:31.410920 19401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:31.412390 19401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab
--principal=oryx/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.107.254:35651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:31.412621 19401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:31.412842 19401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:31.417683 19406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:31.417709 19407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:31.417752 19401 server_base.cc:1034] running on GCE node
W20260926 22:40:31.417909 19409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:31.418129 19401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:31.418406 19401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:31.419546 19401 hybrid_clock.cc:648] HybridClock initialized: now 1790462431419535 us; error 28 us; skew 500 ppm
Sep 26 22:40:31 dist-test-slave-j8sq krb5kdc[19144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462431, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:31.421800 19401 init.cc:377] Logged in from keytab as oryx/127.11.107.194@KRBTEST.COM (short username oryx)
I20260926 22:40:31.422286 19401 webserver.cc:460] Webserver started at http://127.11.107.194:33849/ using document root <none> and password file <none>
I20260926 22:40:31.422501 19401 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:31.422566 19401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:31.422714 19401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:31.423735 19401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "b09f24d99cdf4230a2eb554ab8d27237"
format_stamp: "Formatted at 2026-09-26 22:40:31 on dist-test-slave-j8sq"
server_key: "ed82f8b8d432fb72ab48e904bb17ea09"
server_key_iv: "94c9ba39c99eef9e1b0456b4c30fc3f9"
server_key_version: "encryptionkey@0"
I20260926 22:40:31.424075 19401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "b09f24d99cdf4230a2eb554ab8d27237"
format_stamp: "Formatted at 2026-09-26 22:40:31 on dist-test-slave-j8sq"
server_key: "ed82f8b8d432fb72ab48e904bb17ea09"
server_key_iv: "94c9ba39c99eef9e1b0456b4c30fc3f9"
server_key_version: "encryptionkey@0"
I20260926 22:40:31.425316 19401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:31.426112 19416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:31.426280 19401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:31.426376 19401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "b09f24d99cdf4230a2eb554ab8d27237"
format_stamp: "Formatted at 2026-09-26 22:40:31 on dist-test-slave-j8sq"
server_key: "ed82f8b8d432fb72ab48e904bb17ea09"
server_key_iv: "94c9ba39c99eef9e1b0456b4c30fc3f9"
server_key_version: "encryptionkey@0"
I20260926 22:40:31.426450 19401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:31.433704 19401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:31.435489 19401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:31.435654 19401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:31.436219 19401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:31.436278 19401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:31.436322 19401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:31.436343 19401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 22:40:31 dist-test-slave-j8sq krb5kdc[19144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462431, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
I20260926 22:40:31.443744 19401 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:44097
I20260926 22:40:31.444126 19401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:40:31.444494 19531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:44097 every 8 connection(s)
I20260926 22:40:31.446708 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.436574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:34141 (local address 127.11.107.254:35651)
0926 22:40:31.436641 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.436643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.436939 (+ 296us) server_negotiation.cc:407] Connection header received
0926 22:40:31.437691 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.437693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.437703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.437780 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:31.438747 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.439310 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.441032 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.441247 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.441337 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.444068 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:31.444070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:31.444070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:31.444078 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:31.445524 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.445957 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.445958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.445959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.446012 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.446201 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.446202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.446202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.446302 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:31.446349 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.446574 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.446632 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 22:40:31.446892 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.436580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:35651 (local address 127.11.107.194:34141)
0926 22:40:31.436783 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:31.436833 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:31.437564 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:31.437899 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:31.437904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:31.438313 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:31.438624 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.438631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.439445 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.439447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.440916 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.440920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.441362 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.441362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.441521 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.442129 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:31.442141 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:31.443964 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:31.445601 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:31.445606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:31.445608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:31.445832 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:31.446055 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:31.446055 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:31.446056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:31.446143 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:31.446355 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:31.446359 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:31.446466 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 22:40:31.446540 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 22:40:31.446652 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
I20260926 22:40:31.447407 19532 heartbeater.cc:347] Connected to a master server at 127.11.107.254:35651
I20260926 22:40:31.447486 19532 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:31.447678 19532 heartbeater.cc:511] Master 127.11.107.254:35651 requested a full tablet report, sending...
I20260926 22:40:31.448071 19189 ts_manager.cc:194] Registered new tserver with Master: b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194:44097)
I20260926 22:40:31.448585 19189 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:34141
I20260926 22:40:31.453467 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.448468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:48221 (local address 127.11.107.254:35651)
0926 22:40:31.448507 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.448509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.448580 (+ 71us) server_negotiation.cc:407] Connection header received
0926 22:40:31.448652 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.448653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.448661 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.448733 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:31.449089 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.449675 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.450390 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.450652 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.450704 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.451517 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:31.451518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:31.451519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:31.451523 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:31.452599 (+ 1076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.452883 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.452884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.452884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.452930 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.453073 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.453074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.453074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.453192 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:31.453243 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.453355 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.453412 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:40:31.453475 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.448380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:35651 (local address 127.11.107.194:48221)
0926 22:40:31.448519 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:31.448530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:31.448590 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:31.448739 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:31.448739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:31.448938 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:31.449012 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.449014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.449761 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.449762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.450325 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.450327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.450716 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.450717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.450761 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.451184 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:31.451189 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:31.451449 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:31.452666 (+ 1217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:31.452667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:31.452668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:31.452822 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:31.452985 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:31.452985 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:31.452986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:31.453032 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:31.453248 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:31.453248 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:31.453282 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 22:40:31.453333 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:40:31.453407 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260926 22:40:31.454027 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 19401
I20260926 22:40:31.454133 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:31.454378 11695 external_mini_cluster.cc:1428] Setting key c7a8d292fe18d1588162c32e913dc023
WARNING: no policy specified for oryx/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab.
Entry for principal oryx/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab.
I20260926 22:40:31.476891 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:35651
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab
--principal=oryx/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:31.556176 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:31.556396 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:31.556419 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:31.557628 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:31.557673 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:31.557693 19541 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:40:31.557757 19541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:31.559288 19541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab
--principal=oryx/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.107.254:35651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:31.559545 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:31.559780 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:31.564610 19547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:31.564615 19549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:31.564606 19546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:31.564749 19541 server_base.cc:1034] running on GCE node
I20260926 22:40:31.565088 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:31.565320 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:31.566493 19541 hybrid_clock.cc:648] HybridClock initialized: now 1790462431566486 us; error 34 us; skew 500 ppm
Sep 26 22:40:31 dist-test-slave-j8sq krb5kdc[19144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462431, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:31.568852 19541 init.cc:377] Logged in from keytab as oryx/127.11.107.195@KRBTEST.COM (short username oryx)
I20260926 22:40:31.569334 19541 webserver.cc:460] Webserver started at http://127.11.107.195:40375/ using document root <none> and password file <none>
I20260926 22:40:31.569561 19541 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:31.569629 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:31.569783 19541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:31.570830 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "71c2062e4cc144b1a1822c64ce42fb3b"
format_stamp: "Formatted at 2026-09-26 22:40:31 on dist-test-slave-j8sq"
server_key: "c39f8fbb78f070d42ce15d7023160e9c"
server_key_iv: "9931218cc4af8f58e362e11fddcddc90"
server_key_version: "encryptionkey@0"
I20260926 22:40:31.571195 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "71c2062e4cc144b1a1822c64ce42fb3b"
format_stamp: "Formatted at 2026-09-26 22:40:31 on dist-test-slave-j8sq"
server_key: "c39f8fbb78f070d42ce15d7023160e9c"
server_key_iv: "9931218cc4af8f58e362e11fddcddc90"
server_key_version: "encryptionkey@0"
I20260926 22:40:31.572613 19541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:40:31.573390 19556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:31.573555 19541 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:31.573663 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "71c2062e4cc144b1a1822c64ce42fb3b"
format_stamp: "Formatted at 2026-09-26 22:40:31 on dist-test-slave-j8sq"
server_key: "c39f8fbb78f070d42ce15d7023160e9c"
server_key_iv: "9931218cc4af8f58e362e11fddcddc90"
server_key_version: "encryptionkey@0"
I20260926 22:40:31.573740 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:31.585080 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:31.586839 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:31.586992 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:31.587519 19541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:31.587580 19541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:31.587610 19541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:31.587627 19541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:31.594638 19541 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:33397
I20260926 22:40:31.594710 19671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:33397 every 8 connection(s)
Sep 26 22:40:31 dist-test-slave-j8sq krb5kdc[19144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462431, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
I20260926 22:40:31.594997 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:40:31.598022 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.587859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:53623 (local address 127.11.107.254:35651)
0926 22:40:31.587904 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.587907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.588143 (+ 236us) server_negotiation.cc:407] Connection header received
0926 22:40:31.588659 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.588660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.588682 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.588743 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:31.589704 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.590283 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.591915 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.592201 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.592309 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.595225 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:31.595227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:31.595227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:31.595235 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:31.596647 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.597054 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.597055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.597055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.597111 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.597372 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.597373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.597373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.597490 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:31.597565 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.597856 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.597935 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:40:31.598204 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.587838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:35651 (local address 127.11.107.195:53623)
0926 22:40:31.588007 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:31.588050 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:31.588544 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:31.588859 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:31.588865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:31.589272 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:31.589576 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.589583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.590354 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.590355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.591826 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.591829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.592331 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.592332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.592468 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.593506 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:31.593514 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:31.595125 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:31.596755 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:31.596762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:31.596764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:31.596976 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:31.597186 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:31.597187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:31.597188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:31.597303 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:31.597556 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:31.597563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:31.597698 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 22:40:31.597803 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 22:40:31.597974 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260926 22:40:31.598672 19672 heartbeater.cc:347] Connected to a master server at 127.11.107.254:35651
I20260926 22:40:31.598766 19672 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:31.598932 19672 heartbeater.cc:511] Master 127.11.107.254:35651 requested a full tablet report, sending...
I20260926 22:40:31.599320 19188 ts_manager.cc:194] Registered new tserver with Master: 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397)
I20260926 22:40:31.599840 19188 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:53623
I20260926 22:40:31.601385 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 19541
I20260926 22:40:31.601506 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:31.601692 11695 external_mini_cluster.cc:1428] Setting key e9b5a59152da5afe06cb775a093c24b6
I20260926 22:40:31.602674 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:40:31.605369 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.599697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:42307 (local address 127.11.107.254:35651)
0926 22:40:31.599736 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.599738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.599862 (+ 124us) server_negotiation.cc:407] Connection header received
0926 22:40:31.599958 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.599959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.599968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.600015 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:31.600370 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.600894 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.601811 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.602124 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.602187 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.603068 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:31.603069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:31.603069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:31.603076 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:31.604285 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.604611 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.604612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.604612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.604663 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.604859 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.604860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.604860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.604964 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:31.605041 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.605214 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.605284 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:40:31.605386 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.599625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:35651 (local address 127.11.107.195:42307)
0926 22:40:31.599798 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:31.599807 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:31.599877 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:31.600051 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:31.600052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:31.600240 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:31.600295 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.600296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.600982 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.600983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.601695 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.601698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.602193 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.602194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.602263 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.602678 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:31.602682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:31.602984 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:31.604364 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:31.604365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:31.604366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:31.604531 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:31.604733 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:31.604734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:31.604734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:31.604789 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:31.605062 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:31.605062 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:31.605104 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 22:40:31.605190 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 22:40:31.605286 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
Sep 26 22:40:31 dist-test-slave-j8sq krb5kdc[19144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462431, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 22:40:31 dist-test-slave-j8sq krb5kdc[19144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:31 dist-test-slave-j8sq krb5kdc[19144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 22:40:31.623605 19239 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:31.618503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34048 (local address 127.11.107.254:35651)
0926 22:40:31.618565 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.618567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.618578 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:40:31.618605 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.618607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.618623 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.618714 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:31.619275 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.619753 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.620481 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.620668 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.620741 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.623507 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34048: BlockingRecv error: recv got EOF from 127.0.0.1:34048 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 22:40:31 dist-test-slave-j8sq krb5kdc[19144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462431, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
I20260926 22:40:31.630596 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.624384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34060 (local address 127.11.107.254:35651)
0926 22:40:31.624449 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.624450 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.624528 (+ 78us) server_negotiation.cc:407] Connection header received
0926 22:40:31.624601 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.624602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.624612 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.624677 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:31.625112 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.625566 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.626308 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.626499 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.626565 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.628428 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:31.628429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:31.628429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:31.628435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:31.629667 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.629989 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.629989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.629990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.630037 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:31.630234 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:31.630235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:31.630235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:31.630334 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:31.630404 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.630488 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.630538 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 22:40:31.632175 19188 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:31.633740 19188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:31.640172 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.636146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.11.107.194:44097)
0926 22:40:31.636349 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.636355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.636555 (+ 200us) server_negotiation.cc:407] Connection header received
0926 22:40:31.636604 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.636607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.636625 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.636716 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:31.636970 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.637735 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.639106 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.639783 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.639869 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.639929 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.640035 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.640072 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":104,"threads_started":1}
I20260926 22:40:31.640162 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.636003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:44097 (local address 127.0.0.1:46766)
0926 22:40:31.636357 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:31.636405 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:31.636477 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:31.636800 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:31.636804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:31.636806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:31.636880 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.636882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.637830 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.637831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.639018 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.639021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.639848 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.639849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.639934 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.639948 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 22:40:31.640021 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 22:40:31.640069 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":76,"threads_started":1}
I20260926 22:40:31.640460 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.636239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:40265 (local address 127.0.0.1:42844)
0926 22:40:31.636475 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:31.636486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:31.636544 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:31.636877 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:31.636878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:31.636880 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:31.636949 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.636950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.638291 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.638292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.639297 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.639299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.640231 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.640232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.640296 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.640300 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:31.640329 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 22:40:31.640375 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":54,"threads_started":1}
I20260926 22:40:31.640558 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.636548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.11.107.195:33397)
0926 22:40:31.636713 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.636716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.636843 (+ 127us) server_negotiation.cc:407] Connection header received
0926 22:40:31.636883 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.636885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.636901 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.636978 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:31.637164 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.638246 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.639449 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.640124 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.640203 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.640255 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.640428 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.640470 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260926 22:40:31.640540 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.636378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:33397 (local address 127.0.0.1:50802)
0926 22:40:31.636732 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:31.636742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:31.636790 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:31.637042 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:31.637043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:31.637045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:31.637076 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.637078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.638328 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.638329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.639381 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.639384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.640283 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.640284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.640340 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.640343 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:31.640394 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:40:31.640425 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":45,"threads_started":1}
I20260926 22:40:31.640820 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.636378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.11.107.193:40265)
0926 22:40:31.636573 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.636578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.636588 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:40:31.636627 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.636629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.636773 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.637030 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:31.637075 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.638202 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.639376 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.640107 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.640573 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.640642 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.640668 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.640715 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":83,"threads_started":1}
I20260926 22:40:31.641322 19456 tablet_service.cc:1467] Processing CreateTablet for tablet ec8559164d244bc4987caff3601c5c7f (DEFAULT_TABLE table=test-table [id=5f312032b1c14e96a0a3c71e59c83c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:31.641331 19606 tablet_service.cc:1467] Processing CreateTablet for tablet ec8559164d244bc4987caff3601c5c7f (DEFAULT_TABLE table=test-table [id=5f312032b1c14e96a0a3c71e59c83c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:31.641666 19456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8559164d244bc4987caff3601c5c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:31.641675 19606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8559164d244bc4987caff3601c5c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:31.642448 19326 tablet_service.cc:1467] Processing CreateTablet for tablet ec8559164d244bc4987caff3601c5c7f (DEFAULT_TABLE table=test-table [id=5f312032b1c14e96a0a3c71e59c83c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:31.642772 19326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8559164d244bc4987caff3601c5c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:31.644814 19697 tablet_bootstrap.cc:492] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b: Bootstrap starting.
I20260926 22:40:31.645406 19697 tablet_bootstrap.cc:654] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:31.645763 19697 log.cc:824] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:31.646109 19698 tablet_bootstrap.cc:492] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237: Bootstrap starting.
I20260926 22:40:31.646516 19700 tablet_bootstrap.cc:492] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b: Bootstrap starting.
I20260926 22:40:31.646575 19697 tablet_bootstrap.cc:492] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b: No bootstrap required, opened a new log
I20260926 22:40:31.646636 19697 ts_tablet_manager.cc:1397] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 22:40:31.646731 19698 tablet_bootstrap.cc:654] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:31.647011 19700 tablet_bootstrap.cc:654] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:31.647025 19698 log.cc:824] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:31.647246 19700 log.cc:824] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:31.647828 19698 tablet_bootstrap.cc:492] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237: No bootstrap required, opened a new log
I20260926 22:40:31.647883 19698 ts_tablet_manager.cc:1397] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:31.648866 19697 raft_consensus.cc:348] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.649020 19697 raft_consensus.cc:374] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:31.649051 19697 raft_consensus.cc:729] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c2062e4cc144b1a1822c64ce42fb3b, State: Initialized, Role: FOLLOWER
I20260926 22:40:31.649181 19697 consensus_queue.cc:260] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.649430 19697 ts_tablet_manager.cc:1428] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 22:40:31.649475 19672 heartbeater.cc:503] Master 127.11.107.254:35651 was elected leader, sending a full tablet report...
I20260926 22:40:31.649909 19698 raft_consensus.cc:348] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.650054 19698 raft_consensus.cc:374] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:31.650099 19698 raft_consensus.cc:729] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f24d99cdf4230a2eb554ab8d27237, State: Initialized, Role: FOLLOWER
I20260926 22:40:31.650206 19698 consensus_queue.cc:260] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.650468 19698 ts_tablet_manager.cc:1428] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:40:31.650501 19532 heartbeater.cc:503] Master 127.11.107.254:35651 was elected leader, sending a full tablet report...
I20260926 22:40:31.650838 19700 tablet_bootstrap.cc:492] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b: No bootstrap required, opened a new log
I20260926 22:40:31.650914 19700 ts_tablet_manager.cc:1397] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 22:40:31.652385 19700 raft_consensus.cc:348] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.652503 19704 raft_consensus.cc:482] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:31.652545 19700 raft_consensus.cc:374] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:31.652586 19700 raft_consensus.cc:729] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f02ff9af9642578f2289ddbdd0f08b, State: Initialized, Role: FOLLOWER
I20260926 22:40:31.652642 19704 raft_consensus.cc:504] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.652710 19700 consensus_queue.cc:260] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.652943 19700 ts_tablet_manager.cc:1428] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:40:31.652932 19704 leader_election.cc:288] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8f02ff9af9642578f2289ddbdd0f08b (127.11.107.193:40265), 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397)
I20260926 22:40:31.652992 19392 heartbeater.cc:503] Master 127.11.107.254:35651 was elected leader, sending a full tablet report...
I20260926 22:40:31.656282 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.653157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:57825 (local address 127.11.107.193:40265)
0926 22:40:31.653345 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.653347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.653356 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:31.653377 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.653377 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.653386 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.653447 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:31.653595 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.654262 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.655473 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.655948 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.656001 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.656007 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.656182 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.656208 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260926 22:40:31.656304 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.653090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:40265 (local address 127.11.107.194:57825)
0926 22:40:31.653217 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:31.653230 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:31.653276 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:31.653459 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:31.653460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:31.653461 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:31.653518 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.653519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.654354 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.654355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.655395 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.655397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.656018 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.656019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.656104 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.656109 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:40:31.656172 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 22:40:31.656212 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 22:40:31.656560 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.653547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:54115 (local address 127.11.107.195:33397)
0926 22:40:31.653635 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.653637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.653652 (+ 15us) server_negotiation.cc:407] Connection header received
0926 22:40:31.653671 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.653671 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.653680 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.653732 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:31.653912 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.654541 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.655754 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.656196 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.656255 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.656262 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.656441 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.656486 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 22:40:31.656562 19346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8559164d244bc4987caff3601c5c7f" candidate_uuid: "b09f24d99cdf4230a2eb554ab8d27237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" is_pre_election: true
I20260926 22:40:31.656677 19346 raft_consensus.cc:2454] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f24d99cdf4230a2eb554ab8d27237 in term 0.
I20260926 22:40:31.656797 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.653214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:33397 (local address 127.11.107.194:54115)
0926 22:40:31.653343 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:31.653352 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:31.653388 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:31.653738 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:31.653739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:31.653741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:31.653818 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.653820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.654639 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.654640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.655675 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.655678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.656265 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.656266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.656346 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.656349 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:31.656650 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 22:40:31.656698 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260926 22:40:31.656857 19420 leader_election.cc:302] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f24d99cdf4230a2eb554ab8d27237, e8f02ff9af9642578f2289ddbdd0f08b; no voters:
I20260926 22:40:31.657028 19704 raft_consensus.cc:2781] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:31.657081 19704 raft_consensus.cc:482] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:31.657104 19704 raft_consensus.cc:3037] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:31.657127 19626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8559164d244bc4987caff3601c5c7f" candidate_uuid: "b09f24d99cdf4230a2eb554ab8d27237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" is_pre_election: true
I20260926 22:40:31.657246 19626 raft_consensus.cc:2454] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f24d99cdf4230a2eb554ab8d27237 in term 0.
I20260926 22:40:31.658111 19704 raft_consensus.cc:504] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.658262 19704 leader_election.cc:288] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [CANDIDATE]: Term 1 election: Requested vote from peers e8f02ff9af9642578f2289ddbdd0f08b (127.11.107.193:40265), 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397)
I20260926 22:40:31.658438 19346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8559164d244bc4987caff3601c5c7f" candidate_uuid: "b09f24d99cdf4230a2eb554ab8d27237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f02ff9af9642578f2289ddbdd0f08b"
I20260926 22:40:31.658514 19346 raft_consensus.cc:3037] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:31.659106 19346 raft_consensus.cc:2454] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f24d99cdf4230a2eb554ab8d27237 in term 1.
I20260926 22:40:31.659220 19626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8559164d244bc4987caff3601c5c7f" candidate_uuid: "b09f24d99cdf4230a2eb554ab8d27237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c2062e4cc144b1a1822c64ce42fb3b"
I20260926 22:40:31.659242 19420 leader_election.cc:302] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f24d99cdf4230a2eb554ab8d27237, e8f02ff9af9642578f2289ddbdd0f08b; no voters:
I20260926 22:40:31.659291 19626 raft_consensus.cc:3037] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:31.659363 19704 raft_consensus.cc:2781] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:31.659508 19704 raft_consensus.cc:686] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [term 1 LEADER]: Becoming Leader. State: Replica: b09f24d99cdf4230a2eb554ab8d27237, State: Running, Role: LEADER
I20260926 22:40:31.659592 19704 consensus_queue.cc:237] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.660115 19626 raft_consensus.cc:2454] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f24d99cdf4230a2eb554ab8d27237 in term 1.
I20260926 22:40:31.660254 19188 catalog_manager.cc:5495] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 reported cstate change: term changed from 0 to 1, leader changed from <none> to b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194). New cstate: current_term: 1 leader_uuid: "b09f24d99cdf4230a2eb554ab8d27237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } health_report { overall_health: UNKNOWN } } }
I20260926 22:40:31.667459 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.665471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.11.107.194:44097)
0926 22:40:31.665522 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.665524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.665531 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:31.665557 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.665558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.665570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.665625 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:31.665736 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.666224 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.666908 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.667069 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.667114 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.667233 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:31.667314 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:31.667366 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.667399 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.667413 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:40:31.669126 19626 raft_consensus.cc:1261] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Refusing update from remote peer b09f24d99cdf4230a2eb554ab8d27237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:31.669478 19346 raft_consensus.cc:1261] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Refusing update from remote peer b09f24d99cdf4230a2eb554ab8d27237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:31.669500 19704 consensus_queue.cc:1035] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:31.669729 19704 consensus_queue.cc:1035] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:31.673039 19712 mvcc.cc:205] Tried to move back new op lower bound from 7333734120114049024 to 7333734120078262272. Current Snapshot: MvccSnapshot[applied={T|T < 7333734120114049024}]
I20260926 22:40:31.674542 19188 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:31.674634 19188 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:31.675446 19188 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287: Sending DeleteTablet for 3 replicas of tablet ec8559164d244bc4987caff3601c5c7f
I20260926 22:40:31.675877 19456 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8559164d244bc4987caff3601c5c7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:31 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:46766
I20260926 22:40:31.676088 19188 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 22:40:31.676165 19326 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8559164d244bc4987caff3601c5c7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:31 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:42844
W20260926 22:40:31.676240 19188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:31.676384 19719 tablet_replica.cc:330] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237: stopping tablet replica
I20260926 22:40:31.676476 19719 raft_consensus.cc:2229] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:31.676553 19719 raft_consensus.cc:2258] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:31.676689 19720 tablet_replica.cc:330] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b: stopping tablet replica
I20260926 22:40:31.676785 19720 raft_consensus.cc:2229] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:31.676931 19720 raft_consensus.cc:2258] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:31.678122 19456 tablet_service.cc:1467] Processing CreateTablet for tablet 5511cc62ca664b45a4ca37b30f40f4b1 (DEFAULT_TABLE table=test-table [id=e4df295cb7f44da281b097e2a2b903bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:31.678133 19326 tablet_service.cc:1467] Processing CreateTablet for tablet 5511cc62ca664b45a4ca37b30f40f4b1 (DEFAULT_TABLE table=test-table [id=e4df295cb7f44da281b097e2a2b903bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:31.678267 19326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5511cc62ca664b45a4ca37b30f40f4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:31.678265 19456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5511cc62ca664b45a4ca37b30f40f4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:31.679139 19605 tablet_service.cc:1467] Processing CreateTablet for tablet 5511cc62ca664b45a4ca37b30f40f4b1 (DEFAULT_TABLE table=test-table [id=e4df295cb7f44da281b097e2a2b903bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:31.679258 19605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5511cc62ca664b45a4ca37b30f40f4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:31.679323 19606 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8559164d244bc4987caff3601c5c7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:31 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:50802
I20260926 22:40:31.680339 19719 ts_tablet_manager.cc:1905] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:31.680663 19700 tablet_bootstrap.cc:492] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b: Bootstrap starting.
I20260926 22:40:31.680742 19698 tablet_bootstrap.cc:492] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237: Bootstrap starting.
I20260926 22:40:31.681111 19700 tablet_bootstrap.cc:654] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:31.681171 19698 tablet_bootstrap.cc:654] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:31.681206 19720 ts_tablet_manager.cc:1905] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:31.681504 19697 tablet_bootstrap.cc:492] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b: Bootstrap starting.
I20260926 22:40:31.681931 19697 tablet_bootstrap.cc:654] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:31.682631 19697 tablet_bootstrap.cc:492] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b: No bootstrap required, opened a new log
I20260926 22:40:31.682695 19697 ts_tablet_manager.cc:1397] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:31.682637 19719 ts_tablet_manager.cc:1918] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:31.682809 19719 log.cc:1196] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/ec8559164d244bc4987caff3601c5c7f
I20260926 22:40:31.682818 19720 ts_tablet_manager.cc:1918] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:31.682875 19720 log.cc:1196] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/wal/wals/ec8559164d244bc4987caff3601c5c7f
I20260926 22:40:31.682850 19697 raft_consensus.cc:348] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.682950 19697 raft_consensus.cc:374] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:31.682982 19697 raft_consensus.cc:729] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c2062e4cc144b1a1822c64ce42fb3b, State: Initialized, Role: FOLLOWER
I20260926 22:40:31.683039 19697 consensus_queue.cc:260] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.683148 19697 ts_tablet_manager.cc:1428] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:31.683307 19720 ts_tablet_manager.cc:1939] T ec8559164d244bc4987caff3601c5c7f P e8f02ff9af9642578f2289ddbdd0f08b: Deleting consensus metadata
I20260926 22:40:31.683692 19176 catalog_manager.cc:4841] TS e8f02ff9af9642578f2289ddbdd0f08b (127.11.107.193:40265): tablet ec8559164d244bc4987caff3601c5c7f (table test-table [id=5f312032b1c14e96a0a3c71e59c83c1b]) successfully deleted
I20260926 22:40:31.683761 19719 ts_tablet_manager.cc:1939] T ec8559164d244bc4987caff3601c5c7f P b09f24d99cdf4230a2eb554ab8d27237: Deleting consensus metadata
I20260926 22:40:31.684108 19176 catalog_manager.cc:4841] TS b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194:44097): tablet ec8559164d244bc4987caff3601c5c7f (table test-table [id=5f312032b1c14e96a0a3c71e59c83c1b]) successfully deleted
I20260926 22:40:31.685964 19721 tablet_replica.cc:330] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b: stopping tablet replica
I20260926 22:40:31.686070 19721 raft_consensus.cc:2229] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:31.686180 19721 raft_consensus.cc:2258] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:31.686556 19721 ts_tablet_manager.cc:1905] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:31.686697 19700 tablet_bootstrap.cc:492] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b: No bootstrap required, opened a new log
I20260926 22:40:31.686738 19700 ts_tablet_manager.cc:1397] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 22:40:31.686832 19698 tablet_bootstrap.cc:492] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237: No bootstrap required, opened a new log
I20260926 22:40:31.686872 19698 ts_tablet_manager.cc:1397] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 22:40:31.686874 19700 raft_consensus.cc:348] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.686935 19700 raft_consensus.cc:374] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:31.686956 19700 raft_consensus.cc:729] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f02ff9af9642578f2289ddbdd0f08b, State: Initialized, Role: FOLLOWER
I20260926 22:40:31.687000 19700 consensus_queue.cc:260] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.687012 19698 raft_consensus.cc:348] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.687071 19698 raft_consensus.cc:374] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:31.687075 19700 ts_tablet_manager.cc:1428] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:31.687091 19698 raft_consensus.cc:729] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f24d99cdf4230a2eb554ab8d27237, State: Initialized, Role: FOLLOWER
I20260926 22:40:31.687134 19698 consensus_queue.cc:260] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.687222 19698 ts_tablet_manager.cc:1428] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:31.688040 19721 ts_tablet_manager.cc:1918] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:31.688123 19721 log.cc:1196] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/ec8559164d244bc4987caff3601c5c7f
I20260926 22:40:31.688344 19721 ts_tablet_manager.cc:1939] T ec8559164d244bc4987caff3601c5c7f P 71c2062e4cc144b1a1822c64ce42fb3b: Deleting consensus metadata
I20260926 22:40:31.688766 19176 catalog_manager.cc:4841] TS 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397): tablet ec8559164d244bc4987caff3601c5c7f (table test-table [id=5f312032b1c14e96a0a3c71e59c83c1b]) successfully deleted
W20260926 22:40:31.694871 19533 tablet.cc:2369] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:40:31.803310 19393 tablet.cc:2369] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:40:31.845670 19673 tablet.cc:2369] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:40:31.952912 19705 raft_consensus.cc:482] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:31.953023 19705 raft_consensus.cc:504] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.953516 19705 leader_election.cc:288] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194:44097), 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397)
I20260926 22:40:31.957760 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.954053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:43465 (local address 127.11.107.195:33397)
0926 22:40:31.954127 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.954129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.954163 (+ 34us) server_negotiation.cc:407] Connection header received
0926 22:40:31.954189 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.954190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.954199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.954249 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:31.954447 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.955201 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.956837 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.957387 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.957467 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.957476 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.957640 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.957689 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 22:40:31.957760 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.953815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:45177 (local address 127.11.107.194:44097)
0926 22:40:31.953908 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:31.953911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:31.954011 (+ 100us) server_negotiation.cc:407] Connection header received
0926 22:40:31.954126 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:31.954127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:31.954141 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:31.954217 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:31.954448 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.955202 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.956837 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.957387 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.957467 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.957476 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:31.957640 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 22:40:31.957682 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 22:40:31.957801 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.953741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:44097 (local address 127.11.107.193:45177)
0926 22:40:31.953917 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:31.953929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:31.954033 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:31.954251 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:31.954252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:31.954255 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:31.954363 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.954367 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.955322 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.955323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.956736 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.956739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.957480 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.957481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.957557 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.957563 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 22:40:31.957620 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 22:40:31.957673 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":64,"thread_start_us":85,"threads_started":1}
I20260926 22:40:31.957801 19726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:31.953981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:33397 (local address 127.11.107.193:43465)
0926 22:40:31.954066 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:31.954076 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:31.954121 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:31.954258 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:31.954259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:31.954261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:31.954363 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.954367 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.955333 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.955333 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.956736 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:31.956739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:31.957479 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:31.957480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:31.957560 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:31.957564 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:31.957620 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 22:40:31.957674 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":81,"thread_start_us":38,"threads_started":1}
I20260926 22:40:31.958349 19626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5511cc62ca664b45a4ca37b30f40f4b1" candidate_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" is_pre_election: true
I20260926 22:40:31.958463 19626 raft_consensus.cc:2454] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f02ff9af9642578f2289ddbdd0f08b in term 0.
I20260926 22:40:31.958405 19486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5511cc62ca664b45a4ca37b30f40f4b1" candidate_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f24d99cdf4230a2eb554ab8d27237" is_pre_election: true
I20260926 22:40:31.958529 19486 raft_consensus.cc:2454] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f02ff9af9642578f2289ddbdd0f08b in term 0.
I20260926 22:40:31.958665 19280 leader_election.cc:302] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71c2062e4cc144b1a1822c64ce42fb3b, e8f02ff9af9642578f2289ddbdd0f08b; no voters:
I20260926 22:40:31.958797 19705 raft_consensus.cc:2781] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:31.958838 19705 raft_consensus.cc:482] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:31.958866 19705 raft_consensus.cc:3037] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:31.959532 19705 raft_consensus.cc:504] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.959672 19705 leader_election.cc:288] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [CANDIDATE]: Term 1 election: Requested vote from peers b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194:44097), 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397)
I20260926 22:40:31.959824 19486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5511cc62ca664b45a4ca37b30f40f4b1" candidate_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f24d99cdf4230a2eb554ab8d27237"
I20260926 22:40:31.959856 19626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5511cc62ca664b45a4ca37b30f40f4b1" candidate_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c2062e4cc144b1a1822c64ce42fb3b"
I20260926 22:40:31.959901 19486 raft_consensus.cc:3037] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:31.959939 19626 raft_consensus.cc:3037] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:31.960567 19486 raft_consensus.cc:2454] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f02ff9af9642578f2289ddbdd0f08b in term 1.
I20260926 22:40:31.960577 19626 raft_consensus.cc:2454] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f02ff9af9642578f2289ddbdd0f08b in term 1.
I20260926 22:40:31.960734 19280 leader_election.cc:302] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f24d99cdf4230a2eb554ab8d27237, e8f02ff9af9642578f2289ddbdd0f08b; no voters:
I20260926 22:40:31.960858 19705 raft_consensus.cc:2781] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:31.961009 19705 raft_consensus.cc:686] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [term 1 LEADER]: Becoming Leader. State: Replica: e8f02ff9af9642578f2289ddbdd0f08b, State: Running, Role: LEADER
I20260926 22:40:31.961088 19705 consensus_queue.cc:237] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:31.961946 19189 catalog_manager.cc:5495] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f02ff9af9642578f2289ddbdd0f08b (127.11.107.193). New cstate: current_term: 1 leader_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } health_report { overall_health: UNKNOWN } } }
Sep 26 22:40:32 dist-test-slave-j8sq krb5kdc[19144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462431, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.254@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
I20260926 22:40:32.058122 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.050715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.11.107.254:35651)
0926 22:40:32.050812 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:32.050815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:32.050936 (+ 121us) server_negotiation.cc:407] Connection header received
0926 22:40:32.051043 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:32.051044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:32.051055 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:32.051129 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:32.051601 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.052211 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.053271 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.053454 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.053514 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.055772 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:32.055773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:32.055773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:32.055779 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:32.057007 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:32.057347 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:32.057347 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:32.057348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:32.057394 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:32.057556 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:32.057557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:32.057557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:32.057657 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:32.057706 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:32.057904 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 22:40:32.058009 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 22:40:32.058197 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.050544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:35651 (local address 127.0.0.1:34068)
0926 22:40:32.050852 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:32.050862 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:32.050955 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:32.051134 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:32.051134 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:32.051466 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:32.051509 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.051512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.052339 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.052340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.053178 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.053180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.053526 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.053527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.053570 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.054254 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:32.054259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:32.055694 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:32.057082 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:32.057088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:32.057090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:32.057282 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:32.057451 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:32.057452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:32.057452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:32.057505 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:32.057710 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:32.057715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:32.057770 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 22:40:32.057846 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 22:40:32.058067 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":85,"thread_start_us":220,"threads_started":1}
I20260926 22:40:32.060142 19189 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:34068:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 22:40:32.060366 19189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:32.061764 19456 tablet_service.cc:1467] Processing CreateTablet for tablet dad8e4066e3d470da40dc83e3f899a21 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0eaa2590063942c49766e56676904d7e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:40:32.061818 19326 tablet_service.cc:1467] Processing CreateTablet for tablet dad8e4066e3d470da40dc83e3f899a21 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0eaa2590063942c49766e56676904d7e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:40:32.061828 19605 tablet_service.cc:1467] Processing CreateTablet for tablet dad8e4066e3d470da40dc83e3f899a21 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0eaa2590063942c49766e56676904d7e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:40:32.061954 19456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad8e4066e3d470da40dc83e3f899a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:32.061962 19326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad8e4066e3d470da40dc83e3f899a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:32.062022 19605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad8e4066e3d470da40dc83e3f899a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:32.063264 19697 tablet_bootstrap.cc:492] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b: Bootstrap starting.
I20260926 22:40:32.063426 19698 tablet_bootstrap.cc:492] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237: Bootstrap starting.
I20260926 22:40:32.063426 19700 tablet_bootstrap.cc:492] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b: Bootstrap starting.
I20260926 22:40:32.063771 19697 tablet_bootstrap.cc:654] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:32.063802 19698 tablet_bootstrap.cc:654] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:32.063822 19700 tablet_bootstrap.cc:654] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:32.064457 19698 tablet_bootstrap.cc:492] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237: No bootstrap required, opened a new log
I20260926 22:40:32.064514 19697 tablet_bootstrap.cc:492] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b: No bootstrap required, opened a new log
I20260926 22:40:32.064515 19698 ts_tablet_manager.cc:1397] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:32.064543 19700 tablet_bootstrap.cc:492] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b: No bootstrap required, opened a new log
I20260926 22:40:32.064570 19697 ts_tablet_manager.cc:1397] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:32.064580 19700 ts_tablet_manager.cc:1397] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:32.064639 19698 raft_consensus.cc:348] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.064719 19698 raft_consensus.cc:374] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:32.064742 19698 raft_consensus.cc:729] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f24d99cdf4230a2eb554ab8d27237, State: Initialized, Role: FOLLOWER
I20260926 22:40:32.064706 19697 raft_consensus.cc:348] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.064721 19700 raft_consensus.cc:348] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.064776 19697 raft_consensus.cc:374] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:32.064795 19700 raft_consensus.cc:374] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:32.064800 19697 raft_consensus.cc:729] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c2062e4cc144b1a1822c64ce42fb3b, State: Initialized, Role: FOLLOWER
I20260926 22:40:32.064823 19700 raft_consensus.cc:729] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f02ff9af9642578f2289ddbdd0f08b, State: Initialized, Role: FOLLOWER
I20260926 22:40:32.064792 19698 consensus_queue.cc:260] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.064844 19697 consensus_queue.cc:260] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.064862 19700 consensus_queue.cc:260] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.064889 19698 ts_tablet_manager.cc:1428] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:32.064929 19697 ts_tablet_manager.cc:1428] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:32.064947 19700 ts_tablet_manager.cc:1428] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:32.064994 19704 tablet_replica.cc:439] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } } }
I20260926 22:40:32.065073 19704 tablet_replica.cc:442] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:32.065068 19705 tablet_replica.cc:439] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } } }
I20260926 22:40:32.065127 19705 tablet_replica.cc:442] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:32.065214 19703 tablet_replica.cc:439] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } } }
I20260926 22:40:32.065318 19703 tablet_replica.cc:442] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:32.255620 19705 raft_consensus.cc:482] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:32.255726 19705 raft_consensus.cc:504] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.255900 19705 leader_election.cc:288] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194:44097), 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397)
I20260926 22:40:32.256201 19486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad8e4066e3d470da40dc83e3f899a21" candidate_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f24d99cdf4230a2eb554ab8d27237" is_pre_election: true
I20260926 22:40:32.256294 19486 raft_consensus.cc:2454] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f02ff9af9642578f2289ddbdd0f08b in term 0.
I20260926 22:40:32.256234 19626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad8e4066e3d470da40dc83e3f899a21" candidate_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" is_pre_election: true
I20260926 22:40:32.256363 19626 raft_consensus.cc:2454] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f02ff9af9642578f2289ddbdd0f08b in term 0.
I20260926 22:40:32.256445 19280 leader_election.cc:302] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f24d99cdf4230a2eb554ab8d27237, e8f02ff9af9642578f2289ddbdd0f08b; no voters:
I20260926 22:40:32.256538 19705 raft_consensus.cc:2781] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:32.256577 19705 raft_consensus.cc:482] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:32.256604 19705 raft_consensus.cc:3037] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:32.257225 19705 raft_consensus.cc:504] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.257350 19705 leader_election.cc:288] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [CANDIDATE]: Term 1 election: Requested vote from peers b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194:44097), 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397)
I20260926 22:40:32.257548 19626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad8e4066e3d470da40dc83e3f899a21" candidate_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c2062e4cc144b1a1822c64ce42fb3b"
I20260926 22:40:32.257547 19486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad8e4066e3d470da40dc83e3f899a21" candidate_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f24d99cdf4230a2eb554ab8d27237"
I20260926 22:40:32.257627 19626 raft_consensus.cc:3037] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:32.257627 19486 raft_consensus.cc:3037] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:32.258412 19486 raft_consensus.cc:2454] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f02ff9af9642578f2289ddbdd0f08b in term 1.
I20260926 22:40:32.258412 19626 raft_consensus.cc:2454] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f02ff9af9642578f2289ddbdd0f08b in term 1.
I20260926 22:40:32.258615 19280 leader_election.cc:302] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71c2062e4cc144b1a1822c64ce42fb3b, e8f02ff9af9642578f2289ddbdd0f08b; no voters:
I20260926 22:40:32.258720 19705 raft_consensus.cc:2781] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:32.258805 19705 raft_consensus.cc:686] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [term 1 LEADER]: Becoming Leader. State: Replica: e8f02ff9af9642578f2289ddbdd0f08b, State: Running, Role: LEADER
I20260926 22:40:32.258862 19705 consensus_queue.cc:237] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.259150 19727 tablet_replica.cc:439] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b: TxnStatusTablet state changed. Reason: New leader e8f02ff9af9642578f2289ddbdd0f08b. Latest consensus state: current_term: 1 leader_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } } }
I20260926 22:40:32.259245 19727 tablet_replica.cc:442] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:40:32.259394 19739 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 22:40:32.259506 19739 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 22:40:32.259512 19187 catalog_manager.cc:5495] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f02ff9af9642578f2289ddbdd0f08b (127.11.107.193). New cstate: current_term: 1 leader_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } health_report { overall_health: UNKNOWN } } }
I20260926 22:40:32.294927 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.292070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42852 (local address 127.11.107.193:40265)
0926 22:40:32.292190 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:32.292192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:32.292200 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:40:32.292258 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:32.292259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:32.292287 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:32.292420 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:32.292587 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.293114 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.294013 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.294249 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.294330 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.294537 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:32.294659 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:32.294752 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:32.294806 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 22:40:32.294827 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
I20260926 22:40:32.294919 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.291981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:40265 (local address 127.0.0.1:42852)
0926 22:40:32.292052 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:32.292064 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:32.292115 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:32.292427 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:32.292428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:32.292430 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:40:32.292479 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.292482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.293208 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.293209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.293922 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.293925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.294329 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.294330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.294397 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.294417 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:40:32.294735 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:40:32.294744 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 22:40:32.294800 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 22:40:32.294849 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 22:40:32.296219 19626 raft_consensus.cc:1261] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Refusing update from remote peer e8f02ff9af9642578f2289ddbdd0f08b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:32.296241 19486 raft_consensus.cc:1261] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Refusing update from remote peer e8f02ff9af9642578f2289ddbdd0f08b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:32.296530 19705 consensus_queue.cc:1035] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:32.296684 19727 consensus_queue.cc:1035] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:32.297235 19703 tablet_replica.cc:439] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b: TxnStatusTablet state changed. Reason: New leader e8f02ff9af9642578f2289ddbdd0f08b. Latest consensus state: current_term: 1 leader_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } } }
I20260926 22:40:32.297323 19703 tablet_replica.cc:442] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:32.297376 19704 tablet_replica.cc:439] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237: TxnStatusTablet state changed. Reason: New leader e8f02ff9af9642578f2289ddbdd0f08b. Latest consensus state: current_term: 1 leader_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } } }
I20260926 22:40:32.297461 19704 tablet_replica.cc:442] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:32.298377 19743 mvcc.cc:205] Tried to move back new op lower bound from 7333734122683138048 to 7333734122532593664. Current Snapshot: MvccSnapshot[applied={T|T < 7333734122683138048}]
I20260926 22:40:32.298498 19727 tablet_replica.cc:439] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } } }
I20260926 22:40:32.298584 19727 tablet_replica.cc:442] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:40:32.298635 19727 tablet_replica.cc:439] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } } }
I20260926 22:40:32.298691 19727 tablet_replica.cc:442] T dad8e4066e3d470da40dc83e3f899a21 P e8f02ff9af9642578f2289ddbdd0f08b: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:40:32.298841 19703 tablet_replica.cc:439] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } } }
I20260926 22:40:32.298915 19703 tablet_replica.cc:442] T dad8e4066e3d470da40dc83e3f899a21 P 71c2062e4cc144b1a1822c64ce42fb3b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:32.299506 19704 tablet_replica.cc:439] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } } }
I20260926 22:40:32.299592 19704 tablet_replica.cc:442] T dad8e4066e3d470da40dc83e3f899a21 P b09f24d99cdf4230a2eb554ab8d27237: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:40:32.300312 19746 mvcc.cc:205] Tried to move back new op lower bound from 7333734122683138048 to 7333734122532593664. Current Snapshot: MvccSnapshot[applied={T|T < 7333734122683138048}]
I20260926 22:40:32.302932 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.300302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.11.107.193:40265)
0926 22:40:32.300352 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:32.300353 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:32.300419 (+ 66us) server_negotiation.cc:407] Connection header received
0926 22:40:32.300472 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:32.300473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:32.300484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:32.300537 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:32.300658 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.301115 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.302230 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.302480 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.302557 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.302708 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:32.302739 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:32.302795 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:32.302857 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 22:40:32.302877 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:40:32.308208 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.305540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:39113 (local address 127.11.107.193:40265)
0926 22:40:32.305575 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:32.305577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:32.305715 (+ 138us) server_negotiation.cc:407] Connection header received
0926 22:40:32.305794 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:32.305795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:32.305805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:32.305897 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:32.306031 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.306462 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.307618 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.307794 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.307844 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.308022 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:32.308043 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:32.308079 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:32.308133 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 22:40:32.308145 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:40:32.308303 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.305482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:40265 (local address 127.11.107.193:39113)
0926 22:40:32.305634 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:32.305644 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:32.305741 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:32.305905 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:32.305906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:32.305908 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:40:32.305973 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.305975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.306548 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.306549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.307538 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.307541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.307868 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.307869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.307933 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.307947 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:40:32.308082 (+ 135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:40:32.308083 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:40:32.308132 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:40:32.308176 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260926 22:40:32.310213 19486 raft_consensus.cc:1261] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Refusing update from remote peer e8f02ff9af9642578f2289ddbdd0f08b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:32.310221 19626 raft_consensus.cc:1261] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Refusing update from remote peer e8f02ff9af9642578f2289ddbdd0f08b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:32.310425 19727 consensus_queue.cc:1035] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:32.311337 19756 consensus_queue.cc:1035] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 22:40:32.324288 19187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:32.324409 19187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:32.324963 19187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287: Sending DeleteTablet for 3 replicas of tablet 5511cc62ca664b45a4ca37b30f40f4b1
I20260926 22:40:32.325254 19456 tablet_service.cc:1514] Processing DeleteTablet for tablet 5511cc62ca664b45a4ca37b30f40f4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:32 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:46766
I20260926 22:40:32.325280 19326 tablet_service.cc:1514] Processing DeleteTablet for tablet 5511cc62ca664b45a4ca37b30f40f4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:32 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:42844
I20260926 22:40:32.325383 19605 tablet_service.cc:1514] Processing DeleteTablet for tablet 5511cc62ca664b45a4ca37b30f40f4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:32 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:50802
I20260926 22:40:32.325506 19762 tablet_replica.cc:330] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237: stopping tablet replica
I20260926 22:40:32.325572 19762 raft_consensus.cc:2229] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:32.325613 19762 raft_consensus.cc:2258] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:32.325977 19762 ts_tablet_manager.cc:1905] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:32.327402 19761 tablet_replica.cc:330] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b: stopping tablet replica
I20260926 22:40:32.327497 19761 raft_consensus.cc:2229] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:32.327592 19761 raft_consensus.cc:2258] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:32.327739 19762 ts_tablet_manager.cc:1918] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:32.327801 19762 log.cc:1196] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/5511cc62ca664b45a4ca37b30f40f4b1
I20260926 22:40:32.328052 19762 ts_tablet_manager.cc:1939] T 5511cc62ca664b45a4ca37b30f40f4b1 P b09f24d99cdf4230a2eb554ab8d27237: Deleting consensus metadata
I20260926 22:40:32.328281 19763 tablet_replica.cc:330] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b: stopping tablet replica
I20260926 22:40:32.328352 19763 raft_consensus.cc:2229] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:32.328369 19176 catalog_manager.cc:4841] TS b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194:44097): tablet 5511cc62ca664b45a4ca37b30f40f4b1 (table test-table [id=e4df295cb7f44da281b097e2a2b903bf]) successfully deleted
I20260926 22:40:32.328433 19763 raft_consensus.cc:2258] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:32.328706 19763 ts_tablet_manager.cc:1905] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:32.328811 19761 ts_tablet_manager.cc:1905] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:32.330304 19761 ts_tablet_manager.cc:1918] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:32.330379 19761 log.cc:1196] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/wal/wals/5511cc62ca664b45a4ca37b30f40f4b1
I20260926 22:40:32.330420 19763 ts_tablet_manager.cc:1918] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:32.330484 19763 log.cc:1196] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/5511cc62ca664b45a4ca37b30f40f4b1
I20260926 22:40:32.330643 19761 ts_tablet_manager.cc:1939] T 5511cc62ca664b45a4ca37b30f40f4b1 P e8f02ff9af9642578f2289ddbdd0f08b: Deleting consensus metadata
I20260926 22:40:32.330715 19763 ts_tablet_manager.cc:1939] T 5511cc62ca664b45a4ca37b30f40f4b1 P 71c2062e4cc144b1a1822c64ce42fb3b: Deleting consensus metadata
I20260926 22:40:32.330989 19176 catalog_manager.cc:4841] TS 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397): tablet 5511cc62ca664b45a4ca37b30f40f4b1 (table test-table [id=e4df295cb7f44da281b097e2a2b903bf]) successfully deleted
I20260926 22:40:32.331079 19176 catalog_manager.cc:4841] TS e8f02ff9af9642578f2289ddbdd0f08b (127.11.107.193:40265): tablet 5511cc62ca664b45a4ca37b30f40f4b1 (table test-table [id=e4df295cb7f44da281b097e2a2b903bf]) successfully deleted
Sep 26 22:40:32 dist-test-slave-j8sq krb5kdc[19144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 22:40:32 dist-test-slave-j8sq krb5kdc[19144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:32 dist-test-slave-j8sq krb5kdc[19144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 22:40:32.351577 19239 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:32.345880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.11.107.254:35651)
0926 22:40:32.345932 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:32.345934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:32.346049 (+ 115us) server_negotiation.cc:407] Connection header received
0926 22:40:32.346099 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:32.346100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:32.346112 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:32.346181 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:32.346780 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.347595 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.348444 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.348733 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.348806 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.351513 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34084: BlockingRecv error: recv got EOF from 127.0.0.1:34084 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 22:40:32 dist-test-slave-j8sq krb5kdc[19144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM
I20260926 22:40:32.359929 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.352406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.11.107.254:35651)
0926 22:40:32.352464 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:32.352466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:32.352475 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:32.352542 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:32.352543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:32.352554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:32.352622 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:32.353037 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.353740 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.354462 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.354759 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.354837 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.357191 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:32.357192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:32.357193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:32.357202 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:32.358799 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:32.359127 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:32.359128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:32.359128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:32.359200 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:32.359385 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:32.359386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:32.359387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:32.359538 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:32.359639 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:32.359745 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 22:40:32.359828 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 22:40:32.361554 19187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:32.361732 19187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:32.363150 19456 tablet_service.cc:1467] Processing CreateTablet for tablet 8e85c1d8c3294e5dbb4a5578ec784e91 (DEFAULT_TABLE table=test-table [id=52c930d1db9048ed8da6c9ca97308265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:32.363134 19326 tablet_service.cc:1467] Processing CreateTablet for tablet 8e85c1d8c3294e5dbb4a5578ec784e91 (DEFAULT_TABLE table=test-table [id=52c930d1db9048ed8da6c9ca97308265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:32.363164 19605 tablet_service.cc:1467] Processing CreateTablet for tablet 8e85c1d8c3294e5dbb4a5578ec784e91 (DEFAULT_TABLE table=test-table [id=52c930d1db9048ed8da6c9ca97308265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:32.363274 19456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e85c1d8c3294e5dbb4a5578ec784e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:32.363274 19326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e85c1d8c3294e5dbb4a5578ec784e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:32.363286 19605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e85c1d8c3294e5dbb4a5578ec784e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:32.364558 19700 tablet_bootstrap.cc:492] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b: Bootstrap starting.
I20260926 22:40:32.364558 19698 tablet_bootstrap.cc:492] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237: Bootstrap starting.
I20260926 22:40:32.364916 19698 tablet_bootstrap.cc:654] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:32.364960 19700 tablet_bootstrap.cc:654] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:32.365190 19697 tablet_bootstrap.cc:492] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b: Bootstrap starting.
I20260926 22:40:32.365513 19700 tablet_bootstrap.cc:492] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b: No bootstrap required, opened a new log
I20260926 22:40:32.365564 19697 tablet_bootstrap.cc:654] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:32.365573 19700 ts_tablet_manager.cc:1397] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:32.365689 19700 raft_consensus.cc:348] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.365767 19700 raft_consensus.cc:374] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:32.365794 19700 raft_consensus.cc:729] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f02ff9af9642578f2289ddbdd0f08b, State: Initialized, Role: FOLLOWER
I20260926 22:40:32.365857 19700 consensus_queue.cc:260] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.365957 19700 ts_tablet_manager.cc:1428] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:32.366138 19698 tablet_bootstrap.cc:492] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237: No bootstrap required, opened a new log
I20260926 22:40:32.366187 19698 ts_tablet_manager.cc:1397] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:32.366293 19698 raft_consensus.cc:348] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.366382 19698 raft_consensus.cc:374] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:32.366408 19698 raft_consensus.cc:729] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f24d99cdf4230a2eb554ab8d27237, State: Initialized, Role: FOLLOWER
I20260926 22:40:32.366456 19698 consensus_queue.cc:260] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.366549 19698 ts_tablet_manager.cc:1428] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:32.370302 19697 tablet_bootstrap.cc:492] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b: No bootstrap required, opened a new log
I20260926 22:40:32.370368 19697 ts_tablet_manager.cc:1397] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260926 22:40:32.370515 19697 raft_consensus.cc:348] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.370586 19697 raft_consensus.cc:374] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:32.370620 19697 raft_consensus.cc:729] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c2062e4cc144b1a1822c64ce42fb3b, State: Initialized, Role: FOLLOWER
I20260926 22:40:32.370677 19697 consensus_queue.cc:260] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.370764 19697 ts_tablet_manager.cc:1428] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:32.378463 19705 raft_consensus.cc:482] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:32.378557 19705 raft_consensus.cc:504] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.378742 19705 leader_election.cc:288] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194:44097), 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397)
I20260926 22:40:32.378923 19485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e85c1d8c3294e5dbb4a5578ec784e91" candidate_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f24d99cdf4230a2eb554ab8d27237" is_pre_election: true
I20260926 22:40:32.379022 19485 raft_consensus.cc:2454] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f02ff9af9642578f2289ddbdd0f08b in term 0.
I20260926 22:40:32.379138 19625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e85c1d8c3294e5dbb4a5578ec784e91" candidate_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" is_pre_election: true
I20260926 22:40:32.379191 19280 leader_election.cc:302] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f24d99cdf4230a2eb554ab8d27237, e8f02ff9af9642578f2289ddbdd0f08b; no voters:
I20260926 22:40:32.379238 19625 raft_consensus.cc:2454] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f02ff9af9642578f2289ddbdd0f08b in term 0.
I20260926 22:40:32.379365 19705 raft_consensus.cc:2781] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:32.379405 19705 raft_consensus.cc:482] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:32.379431 19705 raft_consensus.cc:3037] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:32.380033 19705 raft_consensus.cc:504] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.380162 19705 leader_election.cc:288] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [CANDIDATE]: Term 1 election: Requested vote from peers b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194:44097), 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397)
I20260926 22:40:32.380323 19485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e85c1d8c3294e5dbb4a5578ec784e91" candidate_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f24d99cdf4230a2eb554ab8d27237"
I20260926 22:40:32.380316 19625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e85c1d8c3294e5dbb4a5578ec784e91" candidate_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c2062e4cc144b1a1822c64ce42fb3b"
I20260926 22:40:32.380393 19625 raft_consensus.cc:3037] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:32.380414 19485 raft_consensus.cc:3037] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:32.381069 19485 raft_consensus.cc:2454] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f02ff9af9642578f2289ddbdd0f08b in term 1.
I20260926 22:40:32.381222 19625 raft_consensus.cc:2454] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f02ff9af9642578f2289ddbdd0f08b in term 1.
I20260926 22:40:32.381253 19280 leader_election.cc:302] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f24d99cdf4230a2eb554ab8d27237, e8f02ff9af9642578f2289ddbdd0f08b; no voters:
I20260926 22:40:32.381349 19705 raft_consensus.cc:2781] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:32.381397 19705 raft_consensus.cc:686] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [term 1 LEADER]: Becoming Leader. State: Replica: e8f02ff9af9642578f2289ddbdd0f08b, State: Running, Role: LEADER
I20260926 22:40:32.381445 19705 consensus_queue.cc:237] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.382046 19187 catalog_manager.cc:5495] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f02ff9af9642578f2289ddbdd0f08b (127.11.107.193). New cstate: current_term: 1 leader_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } health_report { overall_health: UNKNOWN } } }
I20260926 22:40:32.390977 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.388670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.11.107.193:40265)
0926 22:40:32.388733 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:32.388735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:32.388744 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:32.388767 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:32.388768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:32.388777 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:32.388833 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:32.388938 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.389416 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.390275 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.390485 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.390571 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.390727 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:32.390763 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:32.390816 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:32.390870 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 22:40:32.390890 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 22:40:32.391839 19625 raft_consensus.cc:1261] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Refusing update from remote peer e8f02ff9af9642578f2289ddbdd0f08b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:32.391839 19485 raft_consensus.cc:1261] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Refusing update from remote peer e8f02ff9af9642578f2289ddbdd0f08b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:32.392051 19756 consensus_queue.cc:1035] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:32.392186 19705 consensus_queue.cc:1035] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:32.394815 19187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:32.394907 19187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:32.395615 19187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287: Sending DeleteTablet for 3 replicas of tablet 8e85c1d8c3294e5dbb4a5578ec784e91
I20260926 22:40:32.395792 19326 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e85c1d8c3294e5dbb4a5578ec784e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:32 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:42844
I20260926 22:40:32.395924 19761 tablet_replica.cc:330] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b: stopping tablet replica
I20260926 22:40:32.395917 19605 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e85c1d8c3294e5dbb4a5578ec784e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:32 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:50802
I20260926 22:40:32.395977 19761 raft_consensus.cc:2229] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:32.396039 19761 raft_consensus.cc:2258] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:32.396018 19606 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e85c1d8c3294e5dbb4a5578ec784e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:32 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:50802
I20260926 22:40:32.396114 19456 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e85c1d8c3294e5dbb4a5578ec784e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:32 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:46766
I20260926 22:40:32.396204 19763 tablet_replica.cc:330] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b: stopping tablet replica
I20260926 22:40:32.396250 19763 raft_consensus.cc:2229] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:32.396301 19763 raft_consensus.cc:2258] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:32.396368 19762 tablet_replica.cc:330] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237: stopping tablet replica
I20260926 22:40:32.396427 19762 raft_consensus.cc:2229] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:32.396472 19762 raft_consensus.cc:2258] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:32.396404 19187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:32.396543 19187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:32.396754 19326 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e85c1d8c3294e5dbb4a5578ec784e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:32 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:42844
I20260926 22:40:32.396874 19761 ts_tablet_manager.cc:1905] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:32.397105 19762 ts_tablet_manager.cc:1905] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:32.398118 19456 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0e7acb22f84237b21d34fcc0e460ae (DEFAULT_TABLE table=test-table [id=84cb98b78182423fb2e6d6b5a91153d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:32.398219 19456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0e7acb22f84237b21d34fcc0e460ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:32.398458 19606 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0e7acb22f84237b21d34fcc0e460ae (DEFAULT_TABLE table=test-table [id=84cb98b78182423fb2e6d6b5a91153d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:32.398487 19761 ts_tablet_manager.cc:1918] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:32.398537 19761 log.cc:1196] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-0/wal/wals/8e85c1d8c3294e5dbb4a5578ec784e91
I20260926 22:40:32.398546 19606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0e7acb22f84237b21d34fcc0e460ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:32.398680 19762 ts_tablet_manager.cc:1918] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:32.398730 19762 log.cc:1196] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/8e85c1d8c3294e5dbb4a5578ec784e91
I20260926 22:40:32.398772 19761 ts_tablet_manager.cc:1939] T 8e85c1d8c3294e5dbb4a5578ec784e91 P e8f02ff9af9642578f2289ddbdd0f08b: Deleting consensus metadata
I20260926 22:40:32.398928 19762 ts_tablet_manager.cc:1939] T 8e85c1d8c3294e5dbb4a5578ec784e91 P b09f24d99cdf4230a2eb554ab8d27237: Deleting consensus metadata
I20260926 22:40:32.399267 19176 catalog_manager.cc:4841] TS b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194:44097): tablet 8e85c1d8c3294e5dbb4a5578ec784e91 (table test-table [id=52c930d1db9048ed8da6c9ca97308265]) successfully deleted
I20260926 22:40:32.399748 19697 tablet_bootstrap.cc:492] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b: Bootstrap starting.
I20260926 22:40:32.399849 19326 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0e7acb22f84237b21d34fcc0e460ae (DEFAULT_TABLE table=test-table [id=84cb98b78182423fb2e6d6b5a91153d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:32.399945 19326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0e7acb22f84237b21d34fcc0e460ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:32.400141 19697 tablet_bootstrap.cc:654] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:32.400728 19697 tablet_bootstrap.cc:492] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b: No bootstrap required, opened a new log
I20260926 22:40:32.400787 19697 ts_tablet_manager.cc:1397] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:32.400936 19700 tablet_bootstrap.cc:492] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b: Bootstrap starting.
I20260926 22:40:32.400933 19697 raft_consensus.cc:348] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.400995 19697 raft_consensus.cc:374] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:32.401015 19697 raft_consensus.cc:729] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c2062e4cc144b1a1822c64ce42fb3b, State: Initialized, Role: FOLLOWER
I20260926 22:40:32.401063 19697 consensus_queue.cc:260] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.401160 19697 ts_tablet_manager.cc:1428] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:32.401320 19700 tablet_bootstrap.cc:654] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:32.401948 19700 tablet_bootstrap.cc:492] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b: No bootstrap required, opened a new log
I20260926 22:40:32.401957 19176 catalog_manager.cc:4841] TS e8f02ff9af9642578f2289ddbdd0f08b (127.11.107.193:40265): tablet 8e85c1d8c3294e5dbb4a5578ec784e91 (table test-table [id=52c930d1db9048ed8da6c9ca97308265]) successfully deleted
I20260926 22:40:32.402009 19700 ts_tablet_manager.cc:1397] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
W20260926 22:40:32.402062 19176 catalog_manager.cc:4808] TS e8f02ff9af9642578f2289ddbdd0f08b (127.11.107.193:40265): delete failed for tablet 8e85c1d8c3294e5dbb4a5578ec784e91 because the tablet was not found. No further retry: Not found: Tablet not found: 8e85c1d8c3294e5dbb4a5578ec784e91
I20260926 22:40:32.402151 19700 raft_consensus.cc:348] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.402244 19700 raft_consensus.cc:374] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:32.402272 19700 raft_consensus.cc:729] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f02ff9af9642578f2289ddbdd0f08b, State: Initialized, Role: FOLLOWER
I20260926 22:40:32.402313 19700 consensus_queue.cc:260] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.402338 19698 tablet_bootstrap.cc:492] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237: Bootstrap starting.
I20260926 22:40:32.402405 19700 ts_tablet_manager.cc:1428] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:32.402660 19698 tablet_bootstrap.cc:654] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:32.403148 19698 tablet_bootstrap.cc:492] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237: No bootstrap required, opened a new log
I20260926 22:40:32.403192 19698 ts_tablet_manager.cc:1397] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:32.403290 19698 raft_consensus.cc:348] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.403357 19698 raft_consensus.cc:374] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:32.403375 19698 raft_consensus.cc:729] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f24d99cdf4230a2eb554ab8d27237, State: Initialized, Role: FOLLOWER
I20260926 22:40:32.403412 19698 consensus_queue.cc:260] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.403492 19698 ts_tablet_manager.cc:1428] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:32.404112 19763 ts_tablet_manager.cc:1905] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:32.405454 19763 ts_tablet_manager.cc:1918] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:32.405519 19763 log.cc:1196] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/8e85c1d8c3294e5dbb4a5578ec784e91
I20260926 22:40:32.405769 19763 ts_tablet_manager.cc:1939] T 8e85c1d8c3294e5dbb4a5578ec784e91 P 71c2062e4cc144b1a1822c64ce42fb3b: Deleting consensus metadata
I20260926 22:40:32.406131 19176 catalog_manager.cc:4841] TS 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397): tablet 8e85c1d8c3294e5dbb4a5578ec784e91 (table test-table [id=52c930d1db9048ed8da6c9ca97308265]) successfully deleted
W20260926 22:40:32.406209 19176 catalog_manager.cc:4808] TS 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397): delete failed for tablet 8e85c1d8c3294e5dbb4a5578ec784e91 because the tablet was not found. No further retry: Not found: Tablet not found: 8e85c1d8c3294e5dbb4a5578ec784e91
I20260926 22:40:32.522698 19703 raft_consensus.cc:482] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:32.522796 19703 raft_consensus.cc:504] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.523092 19703 leader_election.cc:288] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8f02ff9af9642578f2289ddbdd0f08b (127.11.107.193:40265), b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194:44097)
I20260926 22:40:32.527245 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.523434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:40029 (local address 127.11.107.193:40265)
0926 22:40:32.523491 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:32.523493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:32.523611 (+ 118us) server_negotiation.cc:407] Connection header received
0926 22:40:32.523693 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:32.523694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:32.523709 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:32.523789 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:32.523969 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.524749 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.526322 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.526878 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.526965 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.526974 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:32.527125 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 22:40:32.527184 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:40:32.527247 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.523657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:38919 (local address 127.11.107.194:44097)
0926 22:40:32.523794 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:32.523797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:32.523806 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:32.523835 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:32.523836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:32.523848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:32.523911 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:32.524070 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.524789 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.526322 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.526902 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.526969 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.526976 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:32.527125 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 22:40:32.527170 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260926 22:40:32.527283 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.523349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:40265 (local address 127.11.107.195:40029)
0926 22:40:32.523474 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:32.523488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:32.523615 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:32.523807 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:32.523808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:32.523811 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:32.523865 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.523866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.524853 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.524854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.526221 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.526224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.526977 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.526978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.527046 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.527050 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:40:32.527107 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 22:40:32.527162 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":65,"thread_start_us":79,"threads_started":1}
I20260926 22:40:32.527285 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.523562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:44097 (local address 127.11.107.195:38919)
0926 22:40:32.523677 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:32.523689 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:32.523751 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:32.523937 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:32.523938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:32.523940 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:32.523996 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.523997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.524889 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.524890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.526221 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.526224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.526979 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.526980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.527051 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.527054 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:32.527107 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 22:40:32.527161 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":52,"thread_start_us":50,"threads_started":1}
I20260926 22:40:32.527568 19485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0e7acb22f84237b21d34fcc0e460ae" candidate_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f24d99cdf4230a2eb554ab8d27237" is_pre_election: true
I20260926 22:40:32.527666 19485 raft_consensus.cc:2454] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c2062e4cc144b1a1822c64ce42fb3b in term 0.
I20260926 22:40:32.527700 19346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0e7acb22f84237b21d34fcc0e460ae" candidate_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" is_pre_election: true
I20260926 22:40:32.527786 19346 raft_consensus.cc:2454] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c2062e4cc144b1a1822c64ce42fb3b in term 0.
I20260926 22:40:32.527818 19560 leader_election.cc:302] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71c2062e4cc144b1a1822c64ce42fb3b, b09f24d99cdf4230a2eb554ab8d27237; no voters:
I20260926 22:40:32.527961 19703 raft_consensus.cc:2781] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:32.527994 19703 raft_consensus.cc:482] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:32.528018 19703 raft_consensus.cc:3037] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:32.528705 19703 raft_consensus.cc:504] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.528858 19703 leader_election.cc:288] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [CANDIDATE]: Term 1 election: Requested vote from peers e8f02ff9af9642578f2289ddbdd0f08b (127.11.107.193:40265), b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194:44097)
I20260926 22:40:32.528988 19346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0e7acb22f84237b21d34fcc0e460ae" candidate_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f02ff9af9642578f2289ddbdd0f08b"
I20260926 22:40:32.529064 19346 raft_consensus.cc:3037] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:32.529021 19485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0e7acb22f84237b21d34fcc0e460ae" candidate_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f24d99cdf4230a2eb554ab8d27237"
I20260926 22:40:32.529127 19485 raft_consensus.cc:3037] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:32.529709 19485 raft_consensus.cc:2454] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c2062e4cc144b1a1822c64ce42fb3b in term 1.
I20260926 22:40:32.529788 19346 raft_consensus.cc:2454] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c2062e4cc144b1a1822c64ce42fb3b in term 1.
I20260926 22:40:32.529917 19560 leader_election.cc:302] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71c2062e4cc144b1a1822c64ce42fb3b, b09f24d99cdf4230a2eb554ab8d27237; no voters:
I20260926 22:40:32.530041 19703 raft_consensus.cc:2781] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:32.530258 19703 raft_consensus.cc:686] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 LEADER]: Becoming Leader. State: Replica: 71c2062e4cc144b1a1822c64ce42fb3b, State: Running, Role: LEADER
I20260926 22:40:32.530339 19703 consensus_queue.cc:237] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } }
I20260926 22:40:32.531023 19187 catalog_manager.cc:5495] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195). New cstate: current_term: 1 leader_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71c2062e4cc144b1a1822c64ce42fb3b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33397 } health_report { overall_health: HEALTHY } } }
I20260926 22:40:32.554836 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.552313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.11.107.195:33397)
0926 22:40:32.552516 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:32.552518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:32.552525 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:32.552568 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:32.552568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:32.552593 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:32.552684 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:32.552794 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.553297 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.554098 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.554322 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.554384 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.554524 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:32.554633 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:32.554699 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:32.554739 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 22:40:32.554758 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260926 22:40:32.559417 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.557385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:35065 (local address 127.11.107.193:40265)
0926 22:40:32.557447 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:32.557449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:32.557479 (+ 30us) server_negotiation.cc:407] Connection header received
0926 22:40:32.557534 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:32.557535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:32.557542 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:32.557589 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:32.557721 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.558193 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.558890 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.559060 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.559107 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.559254 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:32.559276 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:32.559309 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:32.559350 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 22:40:32.559363 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 22:40:32.559437 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.557236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:40265 (local address 127.11.107.195:35065)
0926 22:40:32.557418 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:32.557429 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:32.557492 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:32.557602 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:32.557603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:32.557605 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:40:32.557658 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.557659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.558250 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.558251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.558819 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.558821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.559120 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.559121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.559166 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.559185 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:40:32.559315 (+ 130us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:40:32.559316 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:40:32.559341 (+ 25us) client_negotiation.cc:240] Negotiation successful
0926 22:40:32.559374 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260926 22:40:32.561555 19485 raft_consensus.cc:1261] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Refusing update from remote peer 71c2062e4cc144b1a1822c64ce42fb3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:32.561589 19346 raft_consensus.cc:1261] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Refusing update from remote peer 71c2062e4cc144b1a1822c64ce42fb3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:32.561802 19703 consensus_queue.cc:1035] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f24d99cdf4230a2eb554ab8d27237" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:32.562017 19789 consensus_queue.cc:1035] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f02ff9af9642578f2289ddbdd0f08b" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 40265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:32.569382 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.567132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:48153 (local address 127.11.107.195:33397)
0926 22:40:32.567190 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:32.567191 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:32.567198 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:32.567216 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:32.567217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:32.567224 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:32.567283 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:32.567372 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.567819 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.568566 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.568867 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.568959 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.569108 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:32.569138 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:32.569207 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:32.569316 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 22:40:32.569336 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:40:32.569402 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.566974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:33397 (local address 127.11.107.193:48153)
0926 22:40:32.567018 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:32.567029 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:32.567065 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:32.567273 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:32.567274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:32.567276 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:40:32.567316 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.567318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.567882 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.567882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.568485 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:32.568487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:32.568963 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:32.568964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:32.569032 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:32.569039 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:40:32.569256 (+ 217us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:40:32.569259 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:32.569301 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 22:40:32.569355 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 22:40:32.575536 19189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:32.575668 19189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:32.576360 19189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 951459dc250d4853b29b3cd022ac5287: Sending DeleteTablet for 3 replicas of tablet 4e0e7acb22f84237b21d34fcc0e460ae
I20260926 22:40:32.576586 19326 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e0e7acb22f84237b21d34fcc0e460ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:32 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:42844
I20260926 22:40:32.576613 19456 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e0e7acb22f84237b21d34fcc0e460ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:32 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:46766
I20260926 22:40:32.576687 19761 tablet_replica.cc:330] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b: stopping tablet replica
I20260926 22:40:32.576722 19762 tablet_replica.cc:330] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237: stopping tablet replica
I20260926 22:40:32.576749 19761 raft_consensus.cc:2229] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:32.576764 19762 raft_consensus.cc:2229] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:32.576781 19761 raft_consensus.cc:2258] T 4e0e7acb22f84237b21d34fcc0e460ae P e8f02ff9af9642578f2289ddbdd0f08b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:32.576793 19762 raft_consensus.cc:2258] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:32.576954 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 19261
I20260926 22:40:32.577068 19606 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e0e7acb22f84237b21d34fcc0e460ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:32 UTC) from {username='oryx', principal='oryx/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:50802
I20260926 22:40:32.577142 19763 tablet_replica.cc:330] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b: stopping tablet replica
I20260926 22:40:32.577184 19763 raft_consensus.cc:2229] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:32.577252 19763 raft_consensus.cc:2258] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:32.577510 19762 ts_tablet_manager.cc:1905] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:32.578240 19763 ts_tablet_manager.cc:1905] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:32.579238 19762 ts_tablet_manager.cc:1918] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:32.579290 19762 log.cc:1196] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/4e0e7acb22f84237b21d34fcc0e460ae
I20260926 22:40:32.579528 19762 ts_tablet_manager.cc:1939] T 4e0e7acb22f84237b21d34fcc0e460ae P b09f24d99cdf4230a2eb554ab8d27237: Deleting consensus metadata
I20260926 22:40:32.579802 19763 ts_tablet_manager.cc:1918] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:40:32.579849 19763 log.cc:1196] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/4e0e7acb22f84237b21d34fcc0e460ae
I20260926 22:40:32.579838 19176 catalog_manager.cc:4841] TS b09f24d99cdf4230a2eb554ab8d27237 (127.11.107.194:44097): tablet 4e0e7acb22f84237b21d34fcc0e460ae (table test-table [id=84cb98b78182423fb2e6d6b5a91153d5]) successfully deleted
I20260926 22:40:32.580049 19763 ts_tablet_manager.cc:1939] T 4e0e7acb22f84237b21d34fcc0e460ae P 71c2062e4cc144b1a1822c64ce42fb3b: Deleting consensus metadata
I20260926 22:40:32.580379 19176 catalog_manager.cc:4841] TS 71c2062e4cc144b1a1822c64ce42fb3b (127.11.107.195:33397): tablet 4e0e7acb22f84237b21d34fcc0e460ae (table test-table [id=84cb98b78182423fb2e6d6b5a91153d5]) successfully deleted
W20260926 22:40:32.586836 19176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:40265 (error 108)
I20260926 22:40:32.587073 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 19401
I20260926 22:40:32.587749 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:32.587150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:40265 (local address 127.0.0.1:42886)
0926 22:40:32.587682 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:32.587717 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:40265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":52,"threads_started":1}
W20260926 22:40:32.587854 19176 catalog_manager.cc:4584] TS e8f02ff9af9642578f2289ddbdd0f08b (127.11.107.193:40265): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e0e7acb22f84237b21d34fcc0e460ae: Network error: Client connection negotiation failed: client connection to 127.11.107.193:40265: connect: Connection refused (error 111)
I20260926 22:40:32.592876 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 19541
I20260926 22:40:32.597918 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 19157
2026-09-26T22:40:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2779 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:32 dist-test-slave-j8sq krb5kdc[19803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:32 dist-test-slave-j8sq krb5kdc[19803](info): set up 2 sockets
Sep 26 22:40:32 dist-test-slave-j8sq krb5kdc[19803](info): commencing operation
krb5kdc: starting...
W20260926 22:40:33.639448 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:33 dist-test-slave-j8sq krb5kdc[19803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:40:33.764618 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:40413
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:46439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:40413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:33.845824 19816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:33.846057 19816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:33.846079 19816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:33.847258 19816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:33.847299 19816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:33.847313 19816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:33.847325 19816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:33.847337 19816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:33.848870 19816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:46439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:40413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:40413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:33.849121 19816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:33.849366 19816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:33.854307 19821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:33.854319 19824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:33.854363 19822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:33.854454 19816 server_base.cc:1034] running on GCE node
I20260926 22:40:33.854683 19816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:33.854983 19816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:33.856135 19816 hybrid_clock.cc:648] HybridClock initialized: now 1790462433856121 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T22:40:33Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1263 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:33 dist-test-slave-j8sq krb5kdc[19829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:33 dist-test-slave-j8sq krb5kdc[19829](info): set up 2 sockets
Sep 26 22:40:33 dist-test-slave-j8sq krb5kdc[19829](info): commencing operation
krb5kdc: starting...
W20260926 22:40:34.967396 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:35 dist-test-slave-j8sq krb5kdc[19829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:40:35.084162 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:34823
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:34823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:35.164757 19842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:35.165036 19842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:35.165072 19842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:35.166272 19842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:35.166343 19842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:35.166366 19842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:35.166388 19842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:35.166417 19842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:35.167860 19842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:34823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:34823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:35.168133 19842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:35.168365 19842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:35.173152 19847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:35.173221 19842 server_base.cc:1034] running on GCE node
W20260926 22:40:35.173197 19848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:35.173194 19850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:35.173637 19842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:35.173974 19842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:35.175139 19842 hybrid_clock.cc:648] HybridClock initialized: now 1790462435175132 us; error 38 us; skew 500 ppm
Sep 26 22:40:35 dist-test-slave-j8sq krb5kdc[19829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:35.183773 19842 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:40:35.184252 19842 webserver.cc:460] Webserver started at http://127.11.107.254:44299/ using document root <none> and password file <none>
I20260926 22:40:35.184489 19842 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:35.184556 19842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:35.184710 19842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:35.185763 19842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "4aaeac905f354e5f9f3d7d63be497376"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "9449863c0f61a0255a927900e26e58de"
server_key_iv: "0adc29acc4eb2d45cae71f6b9230fd55"
server_key_version: "encryptionkey@0"
I20260926 22:40:35.186198 19842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "4aaeac905f354e5f9f3d7d63be497376"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "9449863c0f61a0255a927900e26e58de"
server_key_iv: "0adc29acc4eb2d45cae71f6b9230fd55"
server_key_version: "encryptionkey@0"
I20260926 22:40:35.187502 19842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:35.188194 19857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:35.188362 19842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:35.188464 19842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "4aaeac905f354e5f9f3d7d63be497376"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "9449863c0f61a0255a927900e26e58de"
server_key_iv: "0adc29acc4eb2d45cae71f6b9230fd55"
server_key_version: "encryptionkey@0"
I20260926 22:40:35.188544 19842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:35.215016 19842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:35.217484 19842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:35.217654 19842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:35.221562 19842 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:34823
I20260926 22:40:35.221602 19909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:34823 every 8 connection(s)
I20260926 22:40:35.222030 19842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:35.222893 19910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:35.225279 19910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376: Bootstrap starting.
I20260926 22:40:35.225999 19910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:35.226289 19910 log.cc:824] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:35.226949 19910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376: No bootstrap required, opened a new log
I20260926 22:40:35.228469 19910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaeac905f354e5f9f3d7d63be497376" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34823 } }
I20260926 22:40:35.228603 19910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:35.228624 19910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aaeac905f354e5f9f3d7d63be497376, State: Initialized, Role: FOLLOWER
I20260926 22:40:35.228812 19910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaeac905f354e5f9f3d7d63be497376" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34823 } }
I20260926 22:40:35.228873 19910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:35.228933 19910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:35.228978 19910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:35.229111 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 19842
I20260926 22:40:35.229236 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:35.229427 11695 external_mini_cluster.cc:1428] Setting key be63ac16254b8a0f70b8532ac84472f4
I20260926 22:40:35.229626 19910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaeac905f354e5f9f3d7d63be497376" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34823 } }
I20260926 22:40:35.229790 19910 leader_election.cc:302] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aaeac905f354e5f9f3d7d63be497376; no voters:
I20260926 22:40:35.229974 19910 leader_election.cc:288] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:35.230001 19914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:35.230244 19914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 1 LEADER]: Becoming Leader. State: Replica: 4aaeac905f354e5f9f3d7d63be497376, State: Running, Role: LEADER
I20260926 22:40:35.230309 19910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:35.230372 19914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaeac905f354e5f9f3d7d63be497376" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34823 } }
I20260926 22:40:35.230726 19914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aaeac905f354e5f9f3d7d63be497376. Latest consensus state: current_term: 1 leader_uuid: "4aaeac905f354e5f9f3d7d63be497376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaeac905f354e5f9f3d7d63be497376" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34823 } } }
I20260926 22:40:35.230814 19916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aaeac905f354e5f9f3d7d63be497376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaeac905f354e5f9f3d7d63be497376" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34823 } } }
I20260926 22:40:35.230906 19916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:35.230881 19914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:35.231586 19922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:35.232126 19922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:35.233968 19922 catalog_manager.cc:1347] Generated new cluster ID: 59375f2f62d1440394be6c070eb07aa1
I20260926 22:40:35.234019 19922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:40:35 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:35.239784 19922 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:35.240447 19922 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:35.241200 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:35.229906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.11.107.254:34823)
0926 22:40:35.230084 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:35.230093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:35.230127 (+ 34us) server_negotiation.cc:407] Connection header received
0926 22:40:35.230266 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:35.230273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:35.230356 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:35.231321 (+ 965us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:35.231466 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.232837 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.233815 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.234406 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.234498 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:35.237387 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:35.237391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:35.237392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:35.237404 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:35.239543 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:35.239981 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:35.239984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:35.239984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:35.240087 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:35.240326 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:35.240327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:35.240328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:35.240589 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:35.240748 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:35.240873 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 22:40:35.241010 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
I20260926 22:40:35.246503 19922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376: Generated new TSK 0
I20260926 22:40:35.246685 19922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:40:35.272698 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34823
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:35.351855 19936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:35.352097 19936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:35.352150 19936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:35.353310 19936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:35.353376 19936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:35.353420 19936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:35.354852 19936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:35.355105 19936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:35.355329 19936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:35.360260 19942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:35.360258 19941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:35.360333 19944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:35.360678 19936 server_base.cc:1034] running on GCE node
I20260926 22:40:35.360888 19936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:35.361145 19936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:35.362304 19936 hybrid_clock.cc:648] HybridClock initialized: now 1790462435362288 us; error 34 us; skew 500 ppm
Sep 26 22:40:35 dist-test-slave-j8sq krb5kdc[19829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:35.364995 19936 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:40:35.365511 19936 webserver.cc:460] Webserver started at http://127.11.107.193:34357/ using document root <none> and password file <none>
I20260926 22:40:35.365763 19936 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:35.365865 19936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:35.366021 19936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:35.367089 19936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "b2053a8082774b12a17f6eee8bf6b551"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "2588f5597d4143d3829a9ccb4b2d776b"
server_key_iv: "60bff81da604985a0aa265a438aa7e8c"
server_key_version: "encryptionkey@0"
I20260926 22:40:35.367463 19936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "b2053a8082774b12a17f6eee8bf6b551"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "2588f5597d4143d3829a9ccb4b2d776b"
server_key_iv: "60bff81da604985a0aa265a438aa7e8c"
server_key_version: "encryptionkey@0"
I20260926 22:40:35.368809 19936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:35.369603 19951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:35.369810 19936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:35.369930 19936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "b2053a8082774b12a17f6eee8bf6b551"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "2588f5597d4143d3829a9ccb4b2d776b"
server_key_iv: "60bff81da604985a0aa265a438aa7e8c"
server_key_version: "encryptionkey@0"
I20260926 22:40:35.370015 19936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:35.382784 19936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:35.384565 19936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:35.384744 19936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:35.384961 19936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:35.385270 19936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:35.385324 19936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:35.385365 19936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:35.385392 19936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:35.391252 19936 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:41567
I20260926 22:40:35.391317 20064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:41567 every 8 connection(s)
I20260926 22:40:35.391638 19936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
Sep 26 22:40:35 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:35.397766 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 19936
I20260926 22:40:35.397912 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:35.398145 11695 external_mini_cluster.cc:1428] Setting key 0fa2df73576b69f9a8b0b6e161075d41
I20260926 22:40:35.401409 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:35.392400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:60973 (local address 127.11.107.254:34823)
0926 22:40:35.392474 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:35.392476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:35.392724 (+ 248us) server_negotiation.cc:407] Connection header received
0926 22:40:35.393189 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:35.393189 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:35.393198 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:35.393257 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:35.394260 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.394757 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.395718 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.395905 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.395951 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:35.398164 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:35.398166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:35.398167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:35.398182 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:35.399979 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:35.400411 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:35.400413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:35.400413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:35.400477 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:35.400748 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:35.400749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:35.400749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:35.400864 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:35.400935 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:35.401233 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 22:40:35.401319 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 22:40:35.401593 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:35.392392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:60973)
0926 22:40:35.392568 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:35.392611 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:35.393096 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:35.393310 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:35.393313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:35.393968 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:35.394164 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:35.394169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.394844 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.394845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:35.395633 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:35.395635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.395975 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.395975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:35.396074 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:35.396541 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:35.396549 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:35.398068 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:35.400066 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:35.400072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:35.400075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:35.400308 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:35.400547 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:35.400548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:35.400549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:35.400672 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:35.400936 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:35.400943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:35.401085 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 22:40:35.401195 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 22:40:35.401356 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260926 22:40:35.402206 20065 heartbeater.cc:347] Connected to a master server at 127.11.107.254:34823
I20260926 22:40:35.402289 20065 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:35.402483 20065 heartbeater.cc:511] Master 127.11.107.254:34823 requested a full tablet report, sending...
I20260926 22:40:35.403013 19874 ts_manager.cc:194] Registered new tserver with Master: b2053a8082774b12a17f6eee8bf6b551 (127.11.107.193:41567)
I20260926 22:40:35.403630 19874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:60973
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:40:35.422689 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34823
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:35.502676 20070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:35.502866 20070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:35.502887 20070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:35.504043 20070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:35.504083 20070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:35.504136 20070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:35.505635 20070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:35.505880 20070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:35.506093 20070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:35.510949 20075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:35.511018 20070 server_base.cc:1034] running on GCE node
W20260926 22:40:35.510922 20076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:35.510929 20078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:35.511314 20070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:35.511569 20070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:35.512706 20070 hybrid_clock.cc:648] HybridClock initialized: now 1790462435512692 us; error 32 us; skew 500 ppm
Sep 26 22:40:35 dist-test-slave-j8sq krb5kdc[19829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:35.515107 20070 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:40:35.515547 20070 webserver.cc:460] Webserver started at http://127.11.107.194:39377/ using document root <none> and password file <none>
I20260926 22:40:35.515797 20070 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:35.515862 20070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:35.516003 20070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:35.517119 20070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "e236aefc4fa8444aaac39f222d19a1e4"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "9d4347e76c7f4f8178836503bc6fff8e"
server_key_iv: "c78052e8588361aa61227c488e5d18a5"
server_key_version: "encryptionkey@0"
I20260926 22:40:35.517483 20070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "e236aefc4fa8444aaac39f222d19a1e4"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "9d4347e76c7f4f8178836503bc6fff8e"
server_key_iv: "c78052e8588361aa61227c488e5d18a5"
server_key_version: "encryptionkey@0"
I20260926 22:40:35.518944 20070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:35.519719 20085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:35.519887 20070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:35.519989 20070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "e236aefc4fa8444aaac39f222d19a1e4"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "9d4347e76c7f4f8178836503bc6fff8e"
server_key_iv: "c78052e8588361aa61227c488e5d18a5"
server_key_version: "encryptionkey@0"
I20260926 22:40:35.520066 20070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:35.535503 20070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:35.537349 20070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:35.537521 20070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:35.537755 20070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:35.538115 20070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:35.538172 20070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:35.538213 20070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:35.538240 20070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:35.543854 20070 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:40641
I20260926 22:40:35.543900 20198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:40641 every 8 connection(s)
I20260926 22:40:35.544248 20070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:40:35.546989 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 20070
I20260926 22:40:35.547086 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:35.547286 11695 external_mini_cluster.cc:1428] Setting key b7696dcd465565ab52a94f299645d5a4
Sep 26 22:40:35 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:35.553673 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:35.544968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:51869 (local address 127.11.107.254:34823)
0926 22:40:35.545030 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:35.545033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:35.545273 (+ 240us) server_negotiation.cc:407] Connection header received
0926 22:40:35.545712 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:35.545713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:35.545722 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:35.545778 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:35.546677 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.547229 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.548455 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.548663 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.548740 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:35.550845 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:35.550846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:35.550847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:35.550855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:35.552318 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:35.552723 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:35.552724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:35.552724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:35.552778 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:35.553028 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:35.553029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:35.553029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:35.553147 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:35.553217 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:35.553492 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 22:40:35.553569 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:40:35.553848 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:35.544954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:51869)
0926 22:40:35.545122 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:35.545159 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:35.545618 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:35.545832 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:35.545849 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:35.546421 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:35.546601 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:35.546605 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.547322 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.547323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:35.548353 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:35.548355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.548747 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.548748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:35.548873 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:35.549372 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:35.549380 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:35.550757 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:35.552418 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:35.552424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:35.552427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:35.552647 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:35.552852 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:35.552853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:35.552854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:35.552960 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:35.553218 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:35.553224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:35.553360 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 22:40:35.553461 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 22:40:35.553612 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260926 22:40:35.554316 20199 heartbeater.cc:347] Connected to a master server at 127.11.107.254:34823
I20260926 22:40:35.554394 20199 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:35.554579 20199 heartbeater.cc:511] Master 127.11.107.254:34823 requested a full tablet report, sending...
I20260926 22:40:35.555039 19874 ts_manager.cc:194] Registered new tserver with Master: e236aefc4fa8444aaac39f222d19a1e4 (127.11.107.194:40641)
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
I20260926 22:40:35.555528 19874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:51869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:40:35.572431 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34823
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:35.651062 20204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:35.651297 20204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:35.651335 20204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:35.652488 20204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:35.652555 20204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:35.652601 20204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:35.654045 20204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:34823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:35.654313 20204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:35.654533 20204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:35.659554 20209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:35.659523 20210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:35.659529 20212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:35.659585 20204 server_base.cc:1034] running on GCE node
I20260926 22:40:35.659993 20204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:35.660240 20204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:35.661375 20204 hybrid_clock.cc:648] HybridClock initialized: now 1790462435661356 us; error 36 us; skew 500 ppm
Sep 26 22:40:35 dist-test-slave-j8sq krb5kdc[19829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:35.663709 20204 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:40:35.664145 20204 webserver.cc:460] Webserver started at http://127.11.107.195:46497/ using document root <none> and password file <none>
I20260926 22:40:35.664386 20204 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:35.664451 20204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:35.664597 20204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:35.665612 20204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "b8a232ea160a4b2fae01f35348cd22b7"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "d7d4f6c79eff59f238de93a7322cb6ac"
server_key_iv: "fd3476ed13d4bc60d127920c6967c4bb"
server_key_version: "encryptionkey@0"
I20260926 22:40:35.665989 20204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "b8a232ea160a4b2fae01f35348cd22b7"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "d7d4f6c79eff59f238de93a7322cb6ac"
server_key_iv: "fd3476ed13d4bc60d127920c6967c4bb"
server_key_version: "encryptionkey@0"
I20260926 22:40:35.667354 20204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:35.668017 20219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:35.668190 20204 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:35.668284 20204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "b8a232ea160a4b2fae01f35348cd22b7"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "d7d4f6c79eff59f238de93a7322cb6ac"
server_key_iv: "fd3476ed13d4bc60d127920c6967c4bb"
server_key_version: "encryptionkey@0"
I20260926 22:40:35.668363 20204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:35.683810 20204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:35.685695 20204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:35.685887 20204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:35.686144 20204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:35.686473 20204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:35.686529 20204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:35.686568 20204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:35.686594 20204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:35.692495 20204 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:34029
I20260926 22:40:35.692546 20332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:34029 every 8 connection(s)
I20260926 22:40:35.692860 20204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:40:35.697109 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 20204
I20260926 22:40:35.697237 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:35.697444 11695 external_mini_cluster.cc:1428] Setting key fdfedcedb4d573d812f4b98d18069c86
Sep 26 22:40:35 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:35.701828 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:35.693521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:53327 (local address 127.11.107.254:34823)
0926 22:40:35.693567 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:35.693570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:35.693805 (+ 235us) server_negotiation.cc:407] Connection header received
0926 22:40:35.694289 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:35.694290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:35.694300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:35.694352 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:35.695225 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.695715 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.696675 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.696852 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.696902 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:35.699309 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:35.699310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:35.699310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:35.699318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:35.700466 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:35.700868 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:35.700869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:35.700869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:35.700932 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:35.701193 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:35.701195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:35.701195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:35.701315 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:35.701390 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:35.701659 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 22:40:35.701739 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:40:35.702044 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:35.693534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:53327)
0926 22:40:35.693682 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:35.693718 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:35.694207 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:35.694402 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:35.694406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:35.694955 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:35.695145 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:35.695149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.695796 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.695797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:35.696594 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:35.696596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:35.696934 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:35.696935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:35.697087 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:35.697684 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:35.697692 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:35.699213 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:35.700561 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:35.700568 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:35.700570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:35.700790 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:35.700998 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:35.700998 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:35.700999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:35.701108 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:35.701380 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:35.701386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:35.701522 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 22:40:35.701631 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 22:40:35.701779 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":88,"threads_started":1}
I20260926 22:40:35.702423 20333 heartbeater.cc:347] Connected to a master server at 127.11.107.254:34823
I20260926 22:40:35.702533 20333 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:35.702701 20333 heartbeater.cc:511] Master 127.11.107.254:34823 requested a full tablet report, sending...
I20260926 22:40:35.703078 19874 ts_manager.cc:194] Registered new tserver with Master: b8a232ea160a4b2fae01f35348cd22b7 (127.11.107.195:34029)
I20260926 22:40:35.703426 19874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:53327
I20260926 22:40:35.709156 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab.
Entry for principal oryx/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab.
WARNING: no policy specified for oryx/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab.
Entry for principal oryx/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab.
WARNING: no policy specified for oryx/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab.
Entry for principal oryx/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab.
I20260926 22:40:35.781374 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 19936
I20260926 22:40:35.785984 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 20070
I20260926 22:40:35.790477 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 20204
I20260926 22:40:35.794929 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 19842
I20260926 22:40:35.800202 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:34823
--webserver_interface=127.11.107.254
--webserver_port=44299
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:34823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:35.881719 20342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:35.881964 20342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:35.881991 20342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:35.883275 20342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:35.883327 20342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:35.883342 20342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:35.883353 20342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:35.883366 20342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:35.884836 20342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:34823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:34823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=44299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:35.885038 20342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:35.885227 20342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:35.890033 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:35.890075 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:35.890034 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:35.890193 20342 server_base.cc:1034] running on GCE node
I20260926 22:40:35.890502 20342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:35.890797 20342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:35.891989 20342 hybrid_clock.cc:648] HybridClock initialized: now 1790462435891973 us; error 42 us; skew 500 ppm
Sep 26 22:40:35 dist-test-slave-j8sq krb5kdc[19829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:35.894599 20342 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:40:35.895094 20342 webserver.cc:460] Webserver started at http://127.11.107.254:44299/ using document root <none> and password file <none>
I20260926 22:40:35.895334 20342 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:35.895424 20342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:35.896811 20342 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:35.897508 20357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:35.897823 20342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:35.897980 20342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "4aaeac905f354e5f9f3d7d63be497376"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "9449863c0f61a0255a927900e26e58de"
server_key_iv: "0adc29acc4eb2d45cae71f6b9230fd55"
server_key_version: "encryptionkey@0"
I20260926 22:40:35.898324 20342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:35.913672 20342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:35.915537 20342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:35.915709 20342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:35.919929 20342 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:34823
I20260926 22:40:35.919968 20409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:34823 every 8 connection(s)
I20260926 22:40:35.920359 20342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:35.921813 20410 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:40:35.922479 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376: Bootstrap starting.
I20260926 22:40:35.925154 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 20342
I20260926 22:40:35.925490 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:41567
--local_ip_for_outbound_sockets=127.11.107.193
--tserver_master_addrs=127.11.107.254:34823
--webserver_port=34357
--webserver_interface=127.11.107.193
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.107.193
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:40:35.930915 20410 log.cc:824] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:35.932526 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:40:35.932862 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376: Bootstrap complete.
I20260926 22:40:35.935086 20410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaeac905f354e5f9f3d7d63be497376" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34823 } }
I20260926 22:40:35.935304 20410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aaeac905f354e5f9f3d7d63be497376, State: Initialized, Role: FOLLOWER
I20260926 22:40:35.935463 20410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaeac905f354e5f9f3d7d63be497376" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34823 } }
I20260926 22:40:35.935545 20410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:35.935595 20410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:35.935650 20410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:40:35.936599 20410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaeac905f354e5f9f3d7d63be497376" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34823 } }
I20260926 22:40:35.936712 20410 leader_election.cc:302] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aaeac905f354e5f9f3d7d63be497376; no voters:
I20260926 22:40:35.936870 20410 leader_election.cc:288] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 22:40:35.936902 20414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 22:40:35.937022 20414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [term 2 LEADER]: Becoming Leader. State: Replica: 4aaeac905f354e5f9f3d7d63be497376, State: Running, Role: LEADER
I20260926 22:40:35.937125 20410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:35.937145 20414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaeac905f354e5f9f3d7d63be497376" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34823 } }
I20260926 22:40:35.937431 20415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aaeac905f354e5f9f3d7d63be497376. Latest consensus state: current_term: 2 leader_uuid: "4aaeac905f354e5f9f3d7d63be497376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaeac905f354e5f9f3d7d63be497376" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34823 } } }
I20260926 22:40:35.937647 20415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:35.937584 20416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4aaeac905f354e5f9f3d7d63be497376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaeac905f354e5f9f3d7d63be497376" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 34823 } } }
I20260926 22:40:35.937786 20416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:35.938032 20424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:35.938602 20424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:35.938776 20424 catalog_manager.cc:1259] Loaded cluster ID: 59375f2f62d1440394be6c070eb07aa1
I20260926 22:40:35.939069 20424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:35.939880 20424 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:35.940059 20424 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4aaeac905f354e5f9f3d7d63be497376: Loaded TSK: 0
I20260926 22:40:35.940275 20424 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 22:40:36.025723 20412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:36.026013 20412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:36.026051 20412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:36.027238 20412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:36.027304 20412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:36.027347 20412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:36.028824 20412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.193.keytab
--principal=oryx/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:41567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=34357
--tserver_master_addrs=127.11.107.254:34823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:36.029119 20412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:36.029363 20412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:36.035352 20437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:36.035362 20439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:36.035447 20436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:36.035590 20412 server_base.cc:1034] running on GCE node
I20260926 22:40:36.035905 20412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:36.036207 20412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:36.037386 20412 hybrid_clock.cc:648] HybridClock initialized: now 1790462436037372 us; error 29 us; skew 500 ppm
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462436, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:36.040602 20412 init.cc:377] Logged in from keytab as oryx/127.11.107.193@KRBTEST.COM (short username oryx)
I20260926 22:40:36.041052 20412 webserver.cc:460] Webserver started at http://127.11.107.193:34357/ using document root <none> and password file <none>
I20260926 22:40:36.041289 20412 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:36.041373 20412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:36.043205 20412 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:36.044066 20446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:36.044286 20412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:36.044394 20412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "b2053a8082774b12a17f6eee8bf6b551"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "2588f5597d4143d3829a9ccb4b2d776b"
server_key_iv: "60bff81da604985a0aa265a438aa7e8c"
server_key_version: "encryptionkey@0"
I20260926 22:40:36.044793 20412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:36.068120 20412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:36.070407 20412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:36.070570 20412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:36.070847 20412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:36.071151 20412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:36.071206 20412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:36.071247 20412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:36.071283 20412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:36.077355 20412 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:41567
I20260926 22:40:36.077401 20559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:41567 every 8 connection(s)
I20260926 22:40:36.077782 20412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:36.081127 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 20412
I20260926 22:40:36.081457 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:40641
--local_ip_for_outbound_sockets=127.11.107.194
--tserver_master_addrs=127.11.107.254:34823
--webserver_port=39377
--webserver_interface=127.11.107.194
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.107.194
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:36.087468 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:36.078511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:48379)
0926 22:40:36.078718 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:36.078769 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:36.079415 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:36.079855 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:36.079859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:36.080622 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:36.080912 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.080918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.082723 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.082724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.083605 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.083608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.084229 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.084230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.084359 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.084913 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:36.084921 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:36.087258 (+ 2337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
W20260926 22:40:36.087832 20563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:36.078630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:48379 (local address 127.11.107.254:34823)
0926 22:40:36.078856 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:36.078865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:36.078911 (+ 46us) server_negotiation.cc:407] Connection header received
0926 22:40:36.079617 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:36.079623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:36.079668 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:36.079781 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:36.081048 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.082596 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.083722 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.084133 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.084255 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.087369 (+ 3114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:48379: BlockingRecv error: recv got EOF from 127.11.107.193:48379 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":141,"threads_started":1}
W20260926 22:40:36.087864 20560 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:34823 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.107.254:34823: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:36.092415 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:36.088169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:33117)
0926 22:40:36.088291 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:36.088302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:36.088343 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:36.088534 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:36.088535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:36.088678 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:36.088717 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.088718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.089333 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.089335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.089891 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.089893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.090231 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.090232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.090295 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.090723 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:36.090727 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:36.092337 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20260926 22:40:36.092468 20563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:36.088195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:33117 (local address 127.11.107.254:34823)
0926 22:40:36.088425 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:36.088426 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:36.088432 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:40:36.088448 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:36.088448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:36.088457 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:36.088518 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:36.088794 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.089255 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.089969 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.090154 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.090235 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.092405 (+ 2170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:33117: BlockingRecv error: recv got EOF from 127.11.107.193:33117 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":95,"threads_started":1}
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:36.096884 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:36.092868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:58747)
0926 22:40:36.092949 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:36.092958 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:36.092999 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:36.093122 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:36.093122 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:36.093249 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:36.093271 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.093272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.093887 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.093888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.094369 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.094370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.094719 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.094720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.094771 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.095173 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:36.095178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:36.096810 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 22:40:36.096928 20563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:36.092913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:58747 (local address 127.11.107.254:34823)
0926 22:40:36.092946 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:36.092948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:36.092987 (+ 39us) server_negotiation.cc:407] Connection header received
0926 22:40:36.093055 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:36.093056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:36.093065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:36.093118 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:36.093343 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.093783 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.094448 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.094636 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.094719 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.096872 (+ 2153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:58747: BlockingRecv error: recv got EOF from 127.11.107.193:58747 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 22:40:36.097224 20560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 22:40:36.165750 20564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:36.166039 20564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:36.166064 20564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:36.167172 20564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:36.167210 20564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:36.167236 20564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:36.168560 20564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.194.keytab
--principal=oryx/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:40641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=39377
--tserver_master_addrs=127.11.107.254:34823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:36.168741 20564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:36.168907 20564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:36.173671 20571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:36.173681 20570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:36.173681 20573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:36.173717 20564 server_base.cc:1034] running on GCE node
I20260926 22:40:36.174078 20564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:36.174350 20564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:36.175509 20564 hybrid_clock.cc:648] HybridClock initialized: now 1790462436175495 us; error 25 us; skew 500 ppm
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462436, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:36.177901 20564 init.cc:377] Logged in from keytab as oryx/127.11.107.194@KRBTEST.COM (short username oryx)
I20260926 22:40:36.178306 20564 webserver.cc:460] Webserver started at http://127.11.107.194:39377/ using document root <none> and password file <none>
I20260926 22:40:36.178527 20564 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:36.178608 20564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:36.179883 20564 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:36.180406 20580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:36.180557 20564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:36.180675 20564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "e236aefc4fa8444aaac39f222d19a1e4"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "9d4347e76c7f4f8178836503bc6fff8e"
server_key_iv: "c78052e8588361aa61227c488e5d18a5"
server_key_version: "encryptionkey@0"
I20260926 22:40:36.180989 20564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:36.195667 20564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:36.197598 20564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:36.197783 20564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:36.198069 20564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:36.198403 20564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:36.198472 20564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:36.198514 20564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:36.198544 20564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:36.204582 20564 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:40641
I20260926 22:40:36.204633 20693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:40641 every 8 connection(s)
I20260926 22:40:36.205004 20564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:40:36.206939 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 20564
I20260926 22:40:36.207242 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:34029
--local_ip_for_outbound_sockets=127.11.107.195
--tserver_master_addrs=127.11.107.254:34823
--webserver_port=46497
--webserver_interface=127.11.107.195
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.107.195
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:36.213577 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:36.205740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:53705)
0926 22:40:36.205953 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:36.205991 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:36.206542 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:36.206776 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:36.206780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:36.207509 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:36.207713 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.207717 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.208710 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.208711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.209922 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.209925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.210411 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.210411 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.210508 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.211085 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:36.211093 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:36.213432 (+ 2339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":123,"threads_started":1}
W20260926 22:40:36.213650 20563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:36.205750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:53705 (local address 127.11.107.254:34823)
0926 22:40:36.205817 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:36.205820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:36.206088 (+ 268us) server_negotiation.cc:407] Connection header received
0926 22:40:36.206637 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:36.206639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:36.206667 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:36.206722 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:36.207804 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.208575 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.209999 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.210329 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.210404 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.213555 (+ 3151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:53705: BlockingRecv error: recv got EOF from 127.11.107.194:53705 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 22:40:36.213883 20694 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:34823 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.107.254:34823: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:36.219017 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:36.214120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:33119)
0926 22:40:36.214175 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:36.214185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:36.214224 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:36.214375 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:36.214376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:36.214556 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:36.214610 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.214612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.215231 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.215233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.216145 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.216148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.216472 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.216473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.216543 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.217063 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:36.217068 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:36.218945 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 22:40:36.219105 20563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:36.214184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:33119 (local address 127.11.107.254:34823)
0926 22:40:36.214220 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:36.214222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:36.214230 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:40:36.214294 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:36.214294 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:36.214301 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:36.214348 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:36.214691 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.215155 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.216226 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.216398 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.216450 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.219024 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:33119: BlockingRecv error: recv got EOF from 127.11.107.194:33119 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:36.223711 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:36.219221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:41359)
0926 22:40:36.219337 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:36.219346 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:36.219388 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:36.219515 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:36.219516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:36.219655 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:36.219685 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.219686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.220364 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.220365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.221150 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.221152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.221458 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.221459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.221509 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.221950 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:36.221955 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:36.223618 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
W20260926 22:40:36.223742 20563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:36.219281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:41359 (local address 127.11.107.254:34823)
0926 22:40:36.219312 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:36.219314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:36.219395 (+ 81us) server_negotiation.cc:407] Connection header received
0926 22:40:36.219430 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:36.219431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:36.219441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:36.219498 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:36.219843 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.220290 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.221227 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.221396 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.221442 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.223682 (+ 2240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:41359: BlockingRecv error: recv got EOF from 127.11.107.194:41359 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:40:36.223846 20694 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 22:40:36.303545 20697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:36.303894 20697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:36.303929 20697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:36.305931 20697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:36.306003 20697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:36.306042 20697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:36.307622 20697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/krb5kdc/oryx_127.11.107.195.keytab
--principal=oryx/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:34029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=46497
--tserver_master_addrs=127.11.107.254:34823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:36.307875 20697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:36.308130 20697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:36.313421 20704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:36.313395 20706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:36.313408 20703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:36.314323 20697 server_base.cc:1034] running on GCE node
I20260926 22:40:36.314565 20697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:36.314823 20697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:36.315990 20697 hybrid_clock.cc:648] HybridClock initialized: now 1790462436315981 us; error 40 us; skew 500 ppm
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462436, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:36.318771 20697 init.cc:377] Logged in from keytab as oryx/127.11.107.195@KRBTEST.COM (short username oryx)
I20260926 22:40:36.319199 20697 webserver.cc:460] Webserver started at http://127.11.107.195:46497/ using document root <none> and password file <none>
I20260926 22:40:36.319416 20697 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:36.319470 20697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:36.320832 20697 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:36.321448 20713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:36.321666 20697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:36.321754 20697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "b8a232ea160a4b2fae01f35348cd22b7"
format_stamp: "Formatted at 2026-09-26 22:40:35 on dist-test-slave-j8sq"
server_key: "d7d4f6c79eff59f238de93a7322cb6ac"
server_key_iv: "fd3476ed13d4bc60d127920c6967c4bb"
server_key_version: "encryptionkey@0"
I20260926 22:40:36.322161 20697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:36.350666 20697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:36.352733 20697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:36.352846 20697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:36.353041 20697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:36.353369 20697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:36.353403 20697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:36.353426 20697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:36.353438 20697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:36.359879 20697 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:34029
I20260926 22:40:36.359923 20826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:34029 every 8 connection(s)
I20260926 22:40:36.360302 20697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:40:36.362924 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 20697
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 22:40:36.368407 20563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:36.361026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:50683 (local address 127.11.107.254:34823)
0926 22:40:36.361073 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:36.361076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:36.361278 (+ 202us) server_negotiation.cc:407] Connection header received
0926 22:40:36.361833 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:36.361850 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:36.361864 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:36.361919 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:36.362898 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.363623 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.364967 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.365185 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.365264 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.368348 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:50683: BlockingRecv error: recv got EOF from 127.11.107.195:50683 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:40:36.368528 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:36.360994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:50683)
0926 22:40:36.361135 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:36.361186 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:36.361719 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:36.361999 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:36.362004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:36.362582 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:36.362787 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.362791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.363739 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.363741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.364846 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.364850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.365270 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.365271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.365405 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.365985 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:36.365994 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:36.368290 (+ 2296us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
W20260926 22:40:36.368793 20827 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:34823 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.107.254:34823: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
I20260926 22:40:36.370651 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:36.363450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.11.107.254:34823)
0926 22:40:36.363503 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:36.363505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:36.363631 (+ 126us) server_negotiation.cc:407] Connection header received
0926 22:40:36.363727 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:36.363728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:36.363740 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:36.363826 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:36.364439 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.364898 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.365644 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.365960 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.366032 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.367262 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:36.367266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:36.367266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:36.367277 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:36.369207 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:36.369575 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:36.369577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:36.369578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:36.369691 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:36.369940 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:36.369941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:36.369941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:36.370248 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:36.370386 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:36.370514 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 22:40:36.370593 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:36.373502 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:36.369010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:47755)
0926 22:40:36.369070 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:36.369081 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:36.369124 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:36.369265 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:36.369265 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:36.369408 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:36.369462 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.369464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.370343 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.370344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.371160 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.371162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.371599 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.371600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.371660 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.371938 (+ 278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:36.371942 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:36.373423 (+ 1481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 22:40:36.373543 20563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:36.369062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:47755 (local address 127.11.107.254:34823)
0926 22:40:36.369100 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:36.369102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:36.369111 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:36.369165 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:36.369166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:36.369175 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:36.369241 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:36.369553 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.370273 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.371219 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.371532 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.371590 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.373485 (+ 1895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:47755: BlockingRecv error: recv got EOF from 127.11.107.195:47755 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:36 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:36.377632 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:36.373800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:53807)
0926 22:40:36.373869 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:36.373879 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:36.373920 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:36.374034 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:36.374035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:36.374154 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:36.374177 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.374178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.374788 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.374789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.375273 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:36.375275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.375613 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.375614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:36.375664 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.375992 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:36.375996 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:36.377556 (+ 1560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 22:40:36.377665 20563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:36.373877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:53807 (local address 127.11.107.254:34823)
0926 22:40:36.373934 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:36.373936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:36.373944 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:40:36.373966 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:36.373967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:36.373976 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:36.374030 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:36.374249 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.374686 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.375352 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:36.375536 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:36.375615 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:36.377612 (+ 1997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:53807: BlockingRecv error: recv got EOF from 127.11.107.195:53807 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 22:40:36.377751 20827 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 22:40:37 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:37 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:37.102208 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:37.097564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:58027)
0926 22:40:37.097710 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:37.097720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:37.097772 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:37.097952 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:37.097953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:37.098116 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:37.098164 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:37.098165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:37.098779 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:37.098780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:37.099345 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:37.099346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:37.099660 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:37.099661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:37.099703 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:37.100292 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:37.100298 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:37.102130 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
W20260926 22:40:37.102274 20832 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:37.097644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:58027 (local address 127.11.107.254:34823)
0926 22:40:37.097755 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:37.097757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:37.097766 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:37.097818 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:37.097819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:37.097828 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:37.097940 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:37.098229 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:37.098710 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:37.099416 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:37.099595 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:37.099644 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:37.102223 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:58027: BlockingRecv error: recv got EOF from 127.11.107.193:58027 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":57,"threads_started":1}
Sep 26 22:40:37 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:37 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:37.229029 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:37.224293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:45059)
0926 22:40:37.224582 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:37.224592 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:37.224650 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:37.224780 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:37.224781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:37.224933 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:37.224981 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:37.224982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:37.225863 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:37.225864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:37.226455 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:37.226457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:37.226805 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:37.226806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:37.226847 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:37.227333 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:37.227338 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:37.228966 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":139,"threads_started":1}
W20260926 22:40:37.229120 20832 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:37.224383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:45059 (local address 127.11.107.254:34823)
0926 22:40:37.224523 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:37.224525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:37.224653 (+ 128us) server_negotiation.cc:407] Connection header received
0926 22:40:37.224702 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:37.224703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:37.224712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:37.224763 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:37.225040 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:37.225777 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:37.226551 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:37.226740 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:37.226792 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:37.229054 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:45059: BlockingRecv error: recv got EOF from 127.11.107.194:45059 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 26 22:40:37 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:37 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:37.382381 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:37.378112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:52105)
0926 22:40:37.378255 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:37.378264 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:37.378318 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:37.378431 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:37.378432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:37.378584 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:37.378633 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:37.378634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:37.379232 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:37.379233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:37.379797 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:37.379799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:37.380109 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:37.380110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:37.380154 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:37.380565 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:37.380569 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:37.382300 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
W20260926 22:40:37.382416 20832 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:37.378157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:52105 (local address 127.11.107.254:34823)
0926 22:40:37.378225 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:37.378227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:37.378317 (+ 90us) server_negotiation.cc:407] Connection header received
0926 22:40:37.378372 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:37.378373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:37.378385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:37.378425 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:37.378687 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:37.379157 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:37.379870 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:37.380046 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:37.380098 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:37.382369 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:52105: BlockingRecv error: recv got EOF from 127.11.107.195:52105 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 22:40:38 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:38 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:38.107139 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:38.102643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:42271)
0926 22:40:38.102784 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:38.102795 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:38.102866 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:38.103011 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:38.103011 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:38.103164 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:38.103217 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:38.103218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:38.104061 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:38.104062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:38.104581 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:38.104582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:38.104888 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:38.104889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:38.104927 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:38.105362 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:38.105366 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:38.107082 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
W20260926 22:40:38.107245 20836 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:38.102737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:42271 (local address 127.11.107.254:34823)
0926 22:40:38.102874 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:38.102876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:38.102883 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:38.102906 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:38.102906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:38.102915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:38.102992 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:38.103291 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:38.103991 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:38.104649 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:38.104828 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:38.104874 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:38.107171 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:42271: BlockingRecv error: recv got EOF from 127.11.107.193:42271 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":94,"threads_started":1}
Sep 26 22:40:38 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:38 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:38.233982 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:38.229507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:46387)
0926 22:40:38.229640 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:38.229657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:38.229705 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:38.229879 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:38.229880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:38.230036 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:38.230085 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:38.230086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:38.230703 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:38.230704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:38.231231 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:38.231232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:38.231531 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:38.231532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:38.231610 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:38.232104 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:38.232108 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:38.233926 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
W20260926 22:40:38.234117 20836 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:38.229599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:46387 (local address 127.11.107.254:34823)
0926 22:40:38.229678 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:38.229680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:38.229690 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:40:38.229765 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:38.229766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:38.229776 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:38.229820 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:38.230150 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:38.230631 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:38.231295 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:38.231472 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:38.231520 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:38.234035 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:46387: BlockingRecv error: recv got EOF from 127.11.107.194:46387 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 22:40:38 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:38 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:38.387455 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:38.382856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:47257)
0926 22:40:38.383007 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:38.383020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:38.383123 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:38.383292 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:38.383293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:38.383555 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:38.383627 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:38.383629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:38.384242 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:38.384242 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:38.384854 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:38.384856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:38.385180 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:38.385180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:38.385220 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:38.385639 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:38.385643 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:38.387400 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
W20260926 22:40:38.387583 20836 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:38.383005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:47257 (local address 127.11.107.254:34823)
0926 22:40:38.383068 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:38.383070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:38.383107 (+ 37us) server_negotiation.cc:407] Connection header received
0926 22:40:38.383200 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:38.383201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:38.383216 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:38.383278 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:38.383700 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:38.384178 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:38.384932 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:38.385100 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:38.385168 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:38.387503 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:47257: BlockingRecv error: recv got EOF from 127.11.107.195:47257 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 22:40:39 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:39 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:39.112140 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:39.107629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:49987)
0926 22:40:39.107761 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:39.107773 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:39.107845 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:39.107984 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:39.107985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:39.108159 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:39.108207 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:39.108209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:39.108816 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:39.108817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:39.109318 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:39.109320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:39.109721 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:39.109722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:39.109759 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:39.110251 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:39.110275 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:39.112049 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
W20260926 22:40:39.112185 20840 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:39.107664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:49987 (local address 127.11.107.254:34823)
0926 22:40:39.107766 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:39.107768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:39.107823 (+ 55us) server_negotiation.cc:407] Connection header received
0926 22:40:39.107888 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:39.107888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:39.107898 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:39.107951 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:39.108270 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:39.108754 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:39.109415 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:39.109647 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:39.109716 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:39.112112 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:49987: BlockingRecv error: recv got EOF from 127.11.107.193:49987 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":57,"threads_started":1}
Sep 26 22:40:39 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:39 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:39.239574 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:39.234389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:38785)
0926 22:40:39.234544 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:39.234557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:39.234647 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:39.234801 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:39.234802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:39.234980 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:39.235026 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:39.235027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:39.235912 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:39.235914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:39.236529 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:39.236533 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:39.237043 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:39.237044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:39.237106 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:39.237625 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:39.237629 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:39.239515 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
W20260926 22:40:39.239710 20840 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:39.234473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:38785 (local address 127.11.107.254:34823)
0926 22:40:39.234544 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:39.234547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:39.234626 (+ 79us) server_negotiation.cc:407] Connection header received
0926 22:40:39.234727 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:39.234728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:39.234740 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:39.234790 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:39.235110 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:39.235789 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:39.236596 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:39.236925 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:39.237021 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:39.239632 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:38785: BlockingRecv error: recv got EOF from 127.11.107.194:38785 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 22:40:39 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:39 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:39.392258 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:39.387928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:33891)
0926 22:40:39.388121 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:39.388131 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:39.388191 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:39.388290 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:39.388291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:39.388455 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:39.388536 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:39.388538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:39.389162 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:39.389163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:39.389709 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:39.389712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:39.390028 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:39.390029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:39.390069 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:39.390488 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:39.390492 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:39.392189 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
W20260926 22:40:39.392419 20840 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:39.388028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:33891 (local address 127.11.107.254:34823)
0926 22:40:39.388075 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:39.388079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:39.388194 (+ 115us) server_negotiation.cc:407] Connection header received
0926 22:40:39.388235 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:39.388235 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:39.388247 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:39.388284 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:39.388595 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:39.389086 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:39.389782 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:39.389969 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:39.390016 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:39.392323 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:33891: BlockingRecv error: recv got EOF from 127.11.107.195:33891 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 22:40:40 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:40 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:40.116950 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:40.112676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:33483)
0926 22:40:40.112792 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:40.112805 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:40.112879 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:40.113062 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:40.113063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:40.113242 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:40.113293 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:40.113294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:40.113969 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:40.113970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:40.114472 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:40.114473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:40.114778 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:40.114779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:40.114818 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:40.115242 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:40.115246 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:40.116859 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
W20260926 22:40:40.117063 20844 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:40.112695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:33483 (local address 127.11.107.254:34823)
0926 22:40:40.112817 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:40.112819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:40.112856 (+ 37us) server_negotiation.cc:407] Connection header received
0926 22:40:40.112955 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:40.112956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:40.112969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:40.113046 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:40.113360 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:40.113900 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:40.114536 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:40.114720 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:40.114768 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:40.116976 (+ 2208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:33483: BlockingRecv error: recv got EOF from 127.11.107.193:33483 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
Sep 26 22:40:40 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:40 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:40.249970 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:40.240425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:44271)
0926 22:40:40.240611 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:40.240622 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:40.240689 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:40.240821 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:40.240822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:40.240988 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:40.241037 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:40.241038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:40.241653 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:40.241654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:40.242220 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:40.242222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:40.242551 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:40.242552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:40.242590 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:40.248136 (+ 5546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:40.248141 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:40.249904 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":76,"threads_started":1}
W20260926 22:40:40.250056 20844 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:40.240497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:44271 (local address 127.11.107.254:34823)
0926 22:40:40.240609 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:40.240611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:40.240695 (+ 84us) server_negotiation.cc:407] Connection header received
0926 22:40:40.240738 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:40.240739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:40.240750 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:40.240798 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:40.241099 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:40.241583 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:40.242294 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:40.242478 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:40.242530 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:40.249990 (+ 7460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:44271: BlockingRecv error: recv got EOF from 127.11.107.194:44271 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 26 22:40:40 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:40 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:40.397132 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:40.392766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:45321)
0926 22:40:40.392898 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:40.392910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:40.392963 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:40.393092 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:40.393093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:40.393256 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:40.393303 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:40.393304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:40.393953 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:40.393953 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:40.394481 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:40.394482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:40.394825 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:40.394826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:40.394865 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:40.395288 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:40.395292 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:40.397075 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
W20260926 22:40:40.397259 20844 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:40.392844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:45321 (local address 127.11.107.254:34823)
0926 22:40:40.392937 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:40.392943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:40.392954 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:40:40.393009 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:40.393010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:40.393022 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:40.393071 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:40.393366 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:40.393890 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:40.394561 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:40.394758 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:40.394814 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:40.397178 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:45321: BlockingRecv error: recv got EOF from 127.11.107.195:45321 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 22:40:41 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:41 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:41.122412 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:41.117509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:34813)
0926 22:40:41.117686 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:41.117700 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:41.117782 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:41.117941 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:41.117942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:41.118126 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:41.118182 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:41.118184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:41.118866 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:41.118867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:41.119535 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:41.119538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:41.119875 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:41.119876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:41.119916 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:41.120349 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:41.120353 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:41.122295 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":100,"threads_started":1}
W20260926 22:40:41.122458 20848 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:41.117605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:34813 (local address 127.11.107.254:34823)
0926 22:40:41.117777 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:41.117779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:41.117789 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:40:41.117849 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:41.117850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:41.117860 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:41.117916 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:41.118259 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:41.118771 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:41.119613 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:41.119808 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:41.119866 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:41.122380 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:34813: BlockingRecv error: recv got EOF from 127.11.107.193:34813 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
Sep 26 22:40:41 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:41 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:41.255035 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:41.250417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:47687)
0926 22:40:41.250561 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:41.250571 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:41.250619 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:41.250777 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:41.250778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:41.250942 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:41.250990 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:41.250991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:41.251611 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:41.251611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:41.252312 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:41.252329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:41.252679 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:41.252680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:41.252717 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:41.253138 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:41.253142 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:41.254938 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
W20260926 22:40:41.255077 20848 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:41.250545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:47687 (local address 127.11.107.254:34823)
0926 22:40:41.250593 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:41.250595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:41.250605 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:40:41.250698 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:41.250699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:41.250708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:41.250761 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:41.251055 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:41.251547 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:41.252405 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:41.252615 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:41.252668 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:41.255018 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:47687: BlockingRecv error: recv got EOF from 127.11.107.194:47687 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 22:40:41 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:41 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:41.402464 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:41.397593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:35671)
0926 22:40:41.397787 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:41.397798 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:41.397884 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:41.398051 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:41.398052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:41.398244 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:41.398292 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:41.398294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:41.398966 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:41.398967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:41.399506 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:41.399508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:41.399869 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:41.399870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:41.399928 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:41.400440 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:41.400445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:41.402400 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":108,"threads_started":1}
W20260926 22:40:41.402652 20848 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:41.397695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:35671 (local address 127.11.107.254:34823)
0926 22:40:41.397769 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:41.397771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:41.397861 (+ 90us) server_negotiation.cc:407] Connection header received
0926 22:40:41.397958 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:41.397959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:41.397979 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:41.398045 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:41.398376 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:41.398880 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:41.399601 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:41.399796 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:41.399872 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:41.402501 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:35671: BlockingRecv error: recv got EOF from 127.11.107.195:35671 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 22:40:42 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:42 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:42.127727 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:42.122902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:54375)
0926 22:40:42.123055 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:42.123066 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:42.123126 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:42.123280 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:42.123281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:42.123441 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:42.123492 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:42.123493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:42.124119 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:42.124119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:42.124756 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:42.124759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:42.125114 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:42.125114 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:42.125175 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:42.125730 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:42.125736 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:42.127668 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
W20260926 22:40:42.127844 20852 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:42.122953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:54375 (local address 127.11.107.254:34823)
0926 22:40:42.123084 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:42.123086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:42.123110 (+ 24us) server_negotiation.cc:407] Connection header received
0926 22:40:42.123190 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:42.123191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:42.123201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:42.123265 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:42.123557 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:42.124052 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:42.124828 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:42.125018 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:42.125081 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:42.127762 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:54375: BlockingRecv error: recv got EOF from 127.11.107.193:54375 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
Sep 26 22:40:42 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:42 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:42.259819 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:42.255499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:56007)
0926 22:40:42.255662 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:42.255672 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:42.255728 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:42.255847 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:42.255848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:42.256009 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:42.256057 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:42.256059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:42.256664 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:42.256665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:42.257170 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:42.257171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:42.257470 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:42.257471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:42.257509 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:42.258019 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:42.258023 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:42.259763 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
W20260926 22:40:42.259929 20852 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:42.255546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:56007 (local address 127.11.107.254:34823)
0926 22:40:42.255607 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:42.255609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:42.255728 (+ 119us) server_negotiation.cc:407] Connection header received
0926 22:40:42.255770 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:42.255771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:42.255782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:42.255832 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:42.256112 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:42.256590 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:42.257234 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:42.257409 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:42.257458 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:42.259855 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:56007: BlockingRecv error: recv got EOF from 127.11.107.194:56007 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 22:40:42 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:42 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:42.407831 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:42.403016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:58765)
0926 22:40:42.403197 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:42.403209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:42.403280 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:42.403413 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:42.403415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:42.403580 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:42.403630 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:42.403632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:42.404457 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:42.404459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:42.405074 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:42.405077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:42.405402 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:42.405404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:42.405444 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:42.405925 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:42.405930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:42.407771 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
W20260926 22:40:42.407963 20852 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:42.403076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:58765 (local address 127.11.107.254:34823)
0926 22:40:42.403145 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:42.403148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:42.403278 (+ 130us) server_negotiation.cc:407] Connection header received
0926 22:40:42.403327 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:42.403328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:42.403341 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:42.403394 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:42.403726 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:42.404329 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:42.405149 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:42.405336 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:42.405389 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:42.407883 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:58765: BlockingRecv error: recv got EOF from 127.11.107.195:58765 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 22:40:43 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:43 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:43.132650 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:43.128200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:36115)
0926 22:40:43.128341 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:43.128352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:43.128401 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:43.128532 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:43.128533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:43.128703 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:43.128771 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:43.128773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:43.129410 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:43.129410 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:43.129996 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:43.129999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:43.130326 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:43.130326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:43.130363 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:43.130857 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:43.130862 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:43.132592 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
W20260926 22:40:43.132755 20856 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:43.128281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:36115 (local address 127.11.107.254:34823)
0926 22:40:43.128384 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:43.128385 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:43.128393 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:40:43.128467 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:43.128468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:43.128477 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:43.128519 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:43.128842 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:43.129345 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:43.130067 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:43.130265 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:43.130317 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:43.132680 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:36115: BlockingRecv error: recv got EOF from 127.11.107.193:36115 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":49,"threads_started":1}
Sep 26 22:40:43 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:43 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:43.264828 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:43.260297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:38055)
0926 22:40:43.260431 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:43.260448 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:43.260502 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:43.260635 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:43.260636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:43.260804 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:43.260852 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:43.260853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:43.261464 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:43.261465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:43.262052 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:43.262054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:43.262386 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:43.262387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:43.262423 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:43.262924 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:43.262928 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:43.264731 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
W20260926 22:40:43.264873 20856 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:43.260347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:38055 (local address 127.11.107.254:34823)
0926 22:40:43.260416 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:43.260419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:43.260506 (+ 87us) server_negotiation.cc:407] Connection header received
0926 22:40:43.260561 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:43.260562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:43.260575 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:43.260618 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:43.260912 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:43.261401 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:43.262130 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:43.262322 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:43.262378 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:43.264811 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:38055: BlockingRecv error: recv got EOF from 127.11.107.194:38055 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 22:40:43 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:43 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:43.413128 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:43.408451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:39761)
0926 22:40:43.408639 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:43.408651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:43.408744 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:43.408920 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:43.408922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:43.409097 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:43.409142 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:43.409160 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:43.409925 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:43.409927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:43.410455 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:43.410457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:43.410819 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:43.410820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:43.410878 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:43.411305 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:43.411310 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:43.413031 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":106,"threads_started":1}
W20260926 22:40:43.413169 20856 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:43.408527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:39761 (local address 127.11.107.254:34823)
0926 22:40:43.408588 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:43.408590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:43.408726 (+ 136us) server_negotiation.cc:407] Connection header received
0926 22:40:43.408831 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:43.408832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:43.408852 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:43.408914 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:43.409245 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:43.409753 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:43.410539 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:43.410733 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:43.410819 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:43.413112 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:39761: BlockingRecv error: recv got EOF from 127.11.107.195:39761 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 22:40:44 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:44 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:44.138538 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:44.133207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:48533)
0926 22:40:44.133353 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:44.133364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:44.133415 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:44.133636 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:44.133637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:44.133796 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:44.133898 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:44.133901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:44.134946 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:44.134948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:44.135495 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:44.135497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:44.135857 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:44.135858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:44.135898 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:44.136452 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:44.136462 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:44.138469 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
W20260926 22:40:44.138667 20860 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:44.133356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:48533 (local address 127.11.107.254:34823)
0926 22:40:44.133506 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:44.133508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:44.133515 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:44.133538 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:44.133539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:44.133549 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:44.133619 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:44.133967 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:44.134755 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:44.135588 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:44.135783 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:44.135845 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:44.138564 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:48533: BlockingRecv error: recv got EOF from 127.11.107.193:48533 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
Sep 26 22:40:44 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:44 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:44.269954 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:44.265367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:33699)
0926 22:40:44.265517 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:44.265529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:44.265597 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:44.265735 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:44.265736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:44.265941 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:44.265986 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:44.265988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:44.266794 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:44.266795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:44.267316 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:44.267317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:44.267692 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:44.267693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:44.267761 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:44.268181 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:44.268185 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:44.269891 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
W20260926 22:40:44.270013 20860 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:44.265407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:33699 (local address 127.11.107.254:34823)
0926 22:40:44.265468 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:44.265470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:44.265581 (+ 111us) server_negotiation.cc:407] Connection header received
0926 22:40:44.265653 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:44.265654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:44.265667 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:44.265716 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:44.266069 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:44.266709 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:44.267410 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:44.267620 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:44.267694 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:44.269968 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:33699: BlockingRecv error: recv got EOF from 127.11.107.194:33699 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 22:40:44 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:44 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:44.418295 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:44.413694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:46521)
0926 22:40:44.413906 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:44.413917 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:44.413970 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:44.414093 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:44.414094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:44.414261 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:44.414311 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:44.414313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:44.414958 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:44.414959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:44.415744 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:44.415749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:44.416061 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:44.416062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:44.416100 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:44.416529 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:44.416533 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:44.418230 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":111,"threads_started":1}
W20260926 22:40:44.418394 20860 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:44.413905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:46521 (local address 127.11.107.254:34823)
0926 22:40:44.413946 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:44.413949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:44.413963 (+ 14us) server_negotiation.cc:407] Connection header received
0926 22:40:44.414017 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:44.414018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:44.414030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:44.414088 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:44.414375 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:44.414880 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:44.415821 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:44.416000 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:44.416051 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:44.418326 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:46521: BlockingRecv error: recv got EOF from 127.11.107.195:46521 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 26 22:40:45 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:45 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:45.143903 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:45.139071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:48893)
0926 22:40:45.139205 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:45.139225 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:45.139280 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:45.139418 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:45.139419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:45.139596 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:45.139648 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:45.139649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:45.140283 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:45.140284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:45.140815 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:45.140818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:45.141160 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:45.141161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:45.141206 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:45.141895 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:45.141899 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:45.143845 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
W20260926 22:40:45.144039 20864 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:45.139127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:48893 (local address 127.11.107.254:34823)
0926 22:40:45.139256 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:45.139258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:45.139266 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:40:45.139329 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:45.139329 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:45.139340 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:45.139401 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:45.139715 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:45.140211 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:45.140908 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:45.141092 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:45.141141 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:45.143955 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:48893: BlockingRecv error: recv got EOF from 127.11.107.193:48893 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
Sep 26 22:40:45 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:45 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:45.278707 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:45.270494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:59251)
0926 22:40:45.270676 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:45.270688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:45.270757 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:45.270908 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:45.270909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:45.271091 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:45.271135 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:45.271137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:45.271843 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:45.271845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:45.272384 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:45.272386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:45.272759 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:45.272760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:45.272825 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:45.276777 (+ 3952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:45.276782 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:45.278637 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":80,"threads_started":1}
W20260926 22:40:45.278805 20864 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:45.270585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:59251 (local address 127.11.107.254:34823)
0926 22:40:45.270669 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:45.270680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:45.270737 (+ 57us) server_negotiation.cc:407] Connection header received
0926 22:40:45.270823 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:45.270824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:45.270835 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:45.270905 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:45.271242 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:45.271738 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:45.272473 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:45.272670 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:45.272759 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:45.278739 (+ 5980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:59251: BlockingRecv error: recv got EOF from 127.11.107.194:59251 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 22:40:45 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:45 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:45.423404 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:45.418858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:50495)
0926 22:40:45.418997 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:45.419008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:45.419076 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:45.419221 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:45.419222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:45.419379 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:45.419426 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:45.419427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:45.420087 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:45.420087 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:45.420596 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:45.420597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:45.420919 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:45.420919 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:45.420960 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:45.421453 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:45.421461 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:45.423307 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
W20260926 22:40:45.423444 20864 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:45.418930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:50495 (local address 127.11.107.254:34823)
0926 22:40:45.418999 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:45.419001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:45.419078 (+ 77us) server_negotiation.cc:407] Connection header received
0926 22:40:45.419147 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:45.419148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:45.419161 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:45.419202 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:45.419487 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:45.420023 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:45.420675 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:45.420855 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:45.420905 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:45.423384 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:50495: BlockingRecv error: recv got EOF from 127.11.107.195:50495 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 22:40:46 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:46 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:46.149052 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:46.144396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:55475)
0926 22:40:46.144552 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:46.144562 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:46.144609 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:46.144746 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:46.144746 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:46.144903 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:46.144952 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:46.144953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:46.145565 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:46.145565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:46.146138 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:46.146141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:46.146460 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:46.146460 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:46.146506 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:46.147021 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:46.147025 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:46.148984 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
W20260926 22:40:46.149137 20868 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:46.144454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:55475 (local address 127.11.107.254:34823)
0926 22:40:46.144587 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:46.144589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:46.144598 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:46.144681 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:46.144682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:46.144692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:46.144742 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:46.145013 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:46.145499 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:46.146210 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:46.146395 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:46.146444 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:46.149051 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:55475: BlockingRecv error: recv got EOF from 127.11.107.193:55475 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
Sep 26 22:40:46 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:46 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:46.286980 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:46.279144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:47117)
0926 22:40:46.279286 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:46.279301 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:46.279353 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:46.279467 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:46.279467 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:46.279643 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:46.279688 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:46.279689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:46.280299 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:46.280300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:46.280811 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:46.280812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:46.281120 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:46.281121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:46.281159 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:46.285030 (+ 3871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:46.285036 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:46.286919 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
W20260926 22:40:46.287122 20868 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:46.279225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:47117 (local address 127.11.107.254:34823)
0926 22:40:46.279270 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:46.279272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:46.279362 (+ 90us) server_negotiation.cc:407] Connection header received
0926 22:40:46.279405 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:46.279406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:46.279416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:46.279462 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:46.279743 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:46.280235 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:46.280884 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:46.281060 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:46.281108 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:46.287037 (+ 5929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:47117: BlockingRecv error: recv got EOF from 127.11.107.194:47117 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 22:40:46 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:46 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:46.428197 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:46.423878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:41501)
0926 22:40:46.424013 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:46.424026 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:46.424083 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:46.424245 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:46.424246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:46.424410 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:46.424457 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:46.424459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:46.425130 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:46.425131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:46.425641 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:46.425643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:46.426029 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:46.426030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:46.426077 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:46.426472 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:46.426476 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:46.428125 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":99,"threads_started":1}
W20260926 22:40:46.428262 20868 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:46.424017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:41501 (local address 127.11.107.254:34823)
0926 22:40:46.424064 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:46.424067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:46.424075 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:40:46.424164 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:46.424165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:46.424176 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:46.424228 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:46.424520 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:46.425071 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:46.425718 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:46.425946 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:46.426032 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:46.428216 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:41501: BlockingRecv error: recv got EOF from 127.11.107.195:41501 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 22:40:47 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:47 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:47.153992 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:47.149460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:50393)
0926 22:40:47.149603 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:47.149613 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:47.149665 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:47.149814 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:47.149815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:47.149996 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:47.150042 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:47.150044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:47.150667 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:47.150668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:47.151202 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:47.151204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:47.151508 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:47.151509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:47.151547 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:47.152040 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:47.152044 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:47.153887 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
W20260926 22:40:47.154027 20872 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:47.149509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:50393 (local address 127.11.107.254:34823)
0926 22:40:47.149664 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:47.149665 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:47.149675 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:40:47.149702 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:47.149703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:47.149713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:47.149797 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:47.150109 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:47.150602 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:47.151268 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:47.151447 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:47.151494 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:47.153956 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:50393: BlockingRecv error: recv got EOF from 127.11.107.193:50393 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
Sep 26 22:40:47 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:47 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:47.291971 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:47.287500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:42173)
0926 22:40:47.287667 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:47.287693 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:47.287753 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:47.287880 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:47.287881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:47.288048 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:47.288092 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:47.288093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:47.288712 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:47.288713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:47.289231 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:47.289232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:47.289536 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:47.289536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:47.289573 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:47.290064 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:47.290068 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:47.291905 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
W20260926 22:40:47.292091 20872 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:47.287579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:42173 (local address 127.11.107.254:34823)
0926 22:40:47.287674 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:47.287676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:47.287753 (+ 77us) server_negotiation.cc:407] Connection header received
0926 22:40:47.287809 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:47.287810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:47.287820 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:47.287861 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:47.288165 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:47.288637 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:47.289299 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:47.289475 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:47.289523 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:47.292013 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:42173: BlockingRecv error: recv got EOF from 127.11.107.194:42173 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 22:40:47 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:47 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:47.433172 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:47.428740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:44265)
0926 22:40:47.428895 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:47.428906 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:47.428968 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:47.429095 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:47.429096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:47.429250 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:47.429307 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:47.429309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:47.429925 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:47.429927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:47.430445 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:47.430447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:47.430830 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:47.430831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:47.430893 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:47.431323 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:47.431327 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:47.433068 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":111,"threads_started":1}
W20260926 22:40:47.433226 20872 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:47.428814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:44265 (local address 127.11.107.254:34823)
0926 22:40:47.428887 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:47.428888 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:47.428970 (+ 82us) server_negotiation.cc:407] Connection header received
0926 22:40:47.429023 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:47.429023 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:47.429035 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:47.429077 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:47.429363 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:47.429800 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:47.430560 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:47.430748 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:47.430820 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:47.433148 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:44265: BlockingRecv error: recv got EOF from 127.11.107.195:44265 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 22:40:48 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:48 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:48.159026 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:48.154579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:50277)
0926 22:40:48.154732 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:48.154743 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:48.154796 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:48.154954 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:48.154955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:48.155117 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:48.155165 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:48.155167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:48.155791 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:48.155792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:48.156303 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:48.156304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:48.156609 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:48.156609 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:48.156659 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:48.157155 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:48.157161 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:48.158969 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
W20260926 22:40:48.159189 20876 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:48.154716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:50277 (local address 127.11.107.254:34823)
0926 22:40:48.154830 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:48.154831 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:48.154840 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:48.154869 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:48.154870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:48.154880 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:48.154937 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:48.155227 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:48.155730 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:48.156370 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:48.156552 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:48.156601 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:48.159109 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:50277: BlockingRecv error: recv got EOF from 127.11.107.193:50277 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
Sep 26 22:40:48 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:48 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:48.297487 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:48.292498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:40489)
0926 22:40:48.292686 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:48.292700 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:48.292776 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:48.292924 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:48.292925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:48.293123 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:48.293167 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:48.293168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:48.293899 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:48.293900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:48.294409 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:48.294411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:48.294783 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:48.294784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:48.294841 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:48.295329 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:48.295341 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:48.297428 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
W20260926 22:40:48.297575 20876 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:48.292602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:40489 (local address 127.11.107.254:34823)
0926 22:40:48.292691 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:48.292694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:48.292754 (+ 60us) server_negotiation.cc:407] Connection header received
0926 22:40:48.292835 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:48.292836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:48.292849 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:48.292918 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:48.293252 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:48.293763 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:48.294499 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:48.294699 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:48.294782 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:48.297519 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:40489: BlockingRecv error: recv got EOF from 127.11.107.194:40489 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 22:40:48 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:48 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:48.438503 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:48.433597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:34579)
0926 22:40:48.433721 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:48.433734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:48.433795 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:48.434004 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:48.434005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:48.434233 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:48.434277 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:48.434278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:48.434967 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:48.434968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:48.435551 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:48.435553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:48.435904 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:48.435905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:48.435961 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:48.436513 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:48.436519 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:48.438444 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
W20260926 22:40:48.438607 20876 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:48.433678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:34579 (local address 127.11.107.254:34823)
0926 22:40:48.433761 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:48.433762 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:48.433774 (+ 12us) server_negotiation.cc:407] Connection header received
0926 22:40:48.433885 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:48.433886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:48.433902 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:48.433960 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:48.434365 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:48.434879 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:48.435633 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:48.435823 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:48.435894 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:48.438516 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:34579: BlockingRecv error: recv got EOF from 127.11.107.195:34579 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 22:40:49 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:49 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:49.163949 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:49.159619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:41191)
0926 22:40:49.159775 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:49.159786 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:49.159844 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:49.159981 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:49.159982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:49.160143 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:49.160193 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:49.160194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:49.160784 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:49.160785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:49.161286 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:49.161288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:49.161587 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:49.161587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:49.161623 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:49.162169 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:49.162173 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:49.163880 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
W20260926 22:40:49.164073 20880 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:49.159740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:41191 (local address 127.11.107.254:34823)
0926 22:40:49.159864 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:49.159865 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:49.159874 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:49.159895 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:49.159895 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:49.159905 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:49.159963 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:49.160256 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:49.160724 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:49.161356 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:49.161529 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:49.161575 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:49.163966 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:41191: BlockingRecv error: recv got EOF from 127.11.107.193:41191 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
Sep 26 22:40:49 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:49 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:49.302810 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:49.298000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:42781)
0926 22:40:49.298147 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:49.298159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:49.298217 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:49.298381 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:49.298382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:49.298545 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:49.298592 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:49.298593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:49.299254 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:49.299254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:49.299791 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:49.299792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:49.300151 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:49.300152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:49.300188 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:49.300689 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:49.300693 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:49.302751 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
W20260926 22:40:49.302937 20880 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:49.298165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:42781 (local address 127.11.107.254:34823)
0926 22:40:49.298235 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:49.298237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:49.298248 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:40:49.298286 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:49.298287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:49.298299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:49.298369 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:49.298677 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:49.299182 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:49.299858 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:49.300078 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:49.300144 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:49.302852 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:42781: BlockingRecv error: recv got EOF from 127.11.107.194:42781 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 22:40:49 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:49 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:49.443337 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:49.439033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:39685)
0926 22:40:49.439187 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:49.439198 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:49.439259 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:49.439394 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:49.439394 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:49.439557 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:49.439603 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:49.439605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:49.440214 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:49.440215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:49.440735 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:49.440737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:49.441042 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:49.441043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:49.441080 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:49.441521 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:49.441525 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:49.443269 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
W20260926 22:40:49.443403 20880 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:49.439079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:39685 (local address 127.11.107.254:34823)
0926 22:40:49.439129 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:49.439131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:49.439261 (+ 130us) server_negotiation.cc:407] Connection header received
0926 22:40:49.439312 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:49.439313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:49.439325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:49.439365 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:49.439662 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:49.440149 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:49.440804 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:49.440984 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:49.441033 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:49.443355 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:39685: BlockingRecv error: recv got EOF from 127.11.107.195:39685 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 22:40:50 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:50 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:50.169149 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:50.164454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:33807)
0926 22:40:50.164654 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:50.164669 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:50.164739 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:50.164886 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:50.164887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:50.165053 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:50.165100 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:50.165102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:50.165734 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:50.165734 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:50.166294 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:50.166296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:50.166627 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:50.166628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:50.166675 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:50.167139 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:50.167144 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:50.169063 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":99,"threads_started":1}
W20260926 22:40:50.169250 20884 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:50.164588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:33807 (local address 127.11.107.254:34823)
0926 22:40:50.164727 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:50.164728 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:50.164737 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:50.164786 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:50.164786 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:50.164824 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:50.164876 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:50.165169 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:50.165662 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:50.166364 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:50.166561 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:50.166614 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:50.169174 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:33807: BlockingRecv error: recv got EOF from 127.11.107.193:33807 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
Sep 26 22:40:50 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:50 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:50.310926 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:50.303298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:39943)
0926 22:40:50.303485 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:50.303496 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:50.303555 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:50.303677 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:50.303678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:50.303836 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:50.303883 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:50.303885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:50.304503 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:50.304503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:50.305024 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:50.305025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:50.305328 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:50.305329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:50.305367 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:50.308838 (+ 3471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:50.308843 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:50.310859 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":104,"threads_started":1}
W20260926 22:40:50.311014 20884 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:50.303352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:39943 (local address 127.11.107.254:34823)
0926 22:40:50.303414 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:50.303416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:50.303555 (+ 139us) server_negotiation.cc:407] Connection header received
0926 22:40:50.303600 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:50.303600 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:50.303611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:50.303664 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:50.303946 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:50.304430 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:50.305089 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:50.305268 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:50.305317 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:50.310949 (+ 5632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:39943: BlockingRecv error: recv got EOF from 127.11.107.194:39943 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 22:40:50 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:50 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.195@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:50.448364 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:50.443796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.195:33401)
0926 22:40:50.443937 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:50.443949 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:50.444011 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:50.444134 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:50.444134 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:50.444310 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:50.444356 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:50.444358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:50.444995 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:50.444995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:50.445567 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:50.445570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:50.445918 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:50.445919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:50.446000 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:50.446440 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:50.446444 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:50.448302 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
W20260926 22:40:50.448495 20884 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:50.443881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:33401 (local address 127.11.107.254:34823)
0926 22:40:50.443926 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:50.443928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:50.443999 (+ 71us) server_negotiation.cc:407] Connection header received
0926 22:40:50.444054 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:50.444054 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:50.444065 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:50.444116 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:50.444422 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:50.444918 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:50.445647 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:50.445821 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:50.445893 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:50.448413 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.195:33401: BlockingRecv error: recv got EOF from 127.11.107.195:33401 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 22:40:51 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:51 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.193@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:51.174345 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:51.169764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.193:38049)
0926 22:40:51.169898 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:51.169911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:51.169985 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:51.170161 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:51.170162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:51.170356 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:51.170403 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:51.170405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.171033 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.171034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:51.171545 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:51.171546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.171877 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.171878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:51.171915 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:51.172340 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:51.172345 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:51.174224 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":93,"threads_started":1}
W20260926 22:40:51.174402 20888 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:51.169830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:38049 (local address 127.11.107.254:34823)
0926 22:40:51.169958 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:51.169960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:51.169969 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:40:51.170059 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:51.170060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:51.170073 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:51.170137 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:51.170476 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.170965 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.171615 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.171815 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.171864 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:51.174324 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.193:38049: BlockingRecv error: recv got EOF from 127.11.107.193:38049 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":46,"threads_started":1}
Sep 26 22:40:51 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:40:51 dist-test-slave-j8sq krb5kdc[19829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.107.194@KRBTEST.COM for oryx/127.11.107.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:40:51.315631 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:51.311467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:34823 (local address 127.11.107.194:44417)
0926 22:40:51.311593 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:51.311603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:51.311665 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:51.311804 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:51.311804 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:51.311959 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:51.312005 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:51.312007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.312616 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.312617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:51.313132 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:51.313134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.313439 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.313440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:51.313478 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:51.313924 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:51.313929 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:51.315577 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.107.254:34823: Server oryx/127.11.107.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
W20260926 22:40:51.315742 20888 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:51.311517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:44417 (local address 127.11.107.254:34823)
0926 22:40:51.311583 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:51.311585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:51.311658 (+ 73us) server_negotiation.cc:407] Connection header received
0926 22:40:51.311728 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:51.311728 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:51.311738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:51.311787 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:51.312066 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.312554 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.313200 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.313377 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.313425 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:51.315670 (+ 2245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.107.194:44417: BlockingRecv error: recv got EOF from 127.11.107.194:44417 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 22:40:51.373413 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 20412
I20260926 22:40:51.377889 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 20564
I20260926 22:40:51.381716 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 20697
I20260926 22:40:51.385637 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 20342
2026-09-26T22:40:51Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17523 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T22:40:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:51Z Disabled control of system clock
I20260926 22:40:51.412969 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:36311
--webserver_interface=127.11.107.254
--webserver_port=0
--builtin_ntp_servers=127.11.107.212:44689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:36311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 22:40:51.492040 20893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:51.492269 20893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:51.492290 20893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:51.493398 20893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:51.493432 20893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:51.493444 20893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:51.493458 20893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:51.493469 20893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:51.494752 20893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:44689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:36311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:36311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:51.494938 20893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:51.495162 20893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:51.499794 20901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:51.499926 20899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:51.499856 20898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:51.500211 20893 server_base.cc:1034] running on GCE node
I20260926 22:40:51.500463 20893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:51.500756 20893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:51.501894 20893 hybrid_clock.cc:648] HybridClock initialized: now 1790462451501931 us; error 97 us; skew 500 ppm
I20260926 22:40:51.503087 20893 webserver.cc:460] Webserver started at http://127.11.107.254:44885/ using document root <none> and password file <none>
I20260926 22:40:51.503337 20893 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:51.503404 20893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:51.503544 20893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:51.504568 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "d34bc9495b374e8a88b9df200169f51f"
format_stamp: "Formatted at 2026-09-26 22:40:51 on dist-test-slave-j8sq"
server_key: "4c18cc6f859ed224795c746a9e401857"
server_key_iv: "6f5c2681ad6a705050737fe62ed2a193"
server_key_version: "encryptionkey@0"
I20260926 22:40:51.504937 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "d34bc9495b374e8a88b9df200169f51f"
format_stamp: "Formatted at 2026-09-26 22:40:51 on dist-test-slave-j8sq"
server_key: "4c18cc6f859ed224795c746a9e401857"
server_key_iv: "6f5c2681ad6a705050737fe62ed2a193"
server_key_version: "encryptionkey@0"
I20260926 22:40:51.506354 20893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:51.507097 20907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:51.507272 20893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:51.507369 20893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "d34bc9495b374e8a88b9df200169f51f"
format_stamp: "Formatted at 2026-09-26 22:40:51 on dist-test-slave-j8sq"
server_key: "4c18cc6f859ed224795c746a9e401857"
server_key_iv: "6f5c2681ad6a705050737fe62ed2a193"
server_key_version: "encryptionkey@0"
I20260926 22:40:51.507448 20893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:51.525542 20893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:51.525935 20893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:51.526105 20893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:51.529820 20893 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:36311
I20260926 22:40:51.529891 20959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:36311 every 8 connection(s)
I20260926 22:40:51.530269 20893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:51.530951 20960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:51.533502 20960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f: Bootstrap starting.
I20260926 22:40:51.534160 20960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:51.534458 20960 log.cc:824] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:51.535104 20960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f: No bootstrap required, opened a new log
I20260926 22:40:51.536599 20960 raft_consensus.cc:348] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34bc9495b374e8a88b9df200169f51f" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 36311 } }
I20260926 22:40:51.536732 20960 raft_consensus.cc:374] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:51.536756 20960 raft_consensus.cc:729] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34bc9495b374e8a88b9df200169f51f, State: Initialized, Role: FOLLOWER
I20260926 22:40:51.536952 20960 consensus_queue.cc:260] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34bc9495b374e8a88b9df200169f51f" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 36311 } }
I20260926 22:40:51.537045 20960 raft_consensus.cc:388] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:51.537081 20960 raft_consensus.cc:482] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:51.537106 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 20893
I20260926 22:40:51.537132 20960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:51.537190 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:51.537391 11695 external_mini_cluster.cc:1428] Setting key 6632e645afb4f80e53765e40b46a327d
I20260926 22:40:51.537812 20960 raft_consensus.cc:504] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34bc9495b374e8a88b9df200169f51f" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 36311 } }
I20260926 22:40:51.537997 20960 leader_election.cc:302] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d34bc9495b374e8a88b9df200169f51f; no voters:
I20260926 22:40:51.538225 20960 leader_election.cc:288] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:51.538272 20965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:51.538450 20965 raft_consensus.cc:686] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [term 1 LEADER]: Becoming Leader. State: Replica: d34bc9495b374e8a88b9df200169f51f, State: Running, Role: LEADER
I20260926 22:40:51.538583 20960 sys_catalog.cc:564] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:51.538610 20965 consensus_queue.cc:237] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34bc9495b374e8a88b9df200169f51f" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 36311 } }
I20260926 22:40:51.539106 20967 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d34bc9495b374e8a88b9df200169f51f. Latest consensus state: current_term: 1 leader_uuid: "d34bc9495b374e8a88b9df200169f51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34bc9495b374e8a88b9df200169f51f" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 36311 } } }
I20260926 22:40:51.539204 20967 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:51.539232 20966 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d34bc9495b374e8a88b9df200169f51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34bc9495b374e8a88b9df200169f51f" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 36311 } } }
I20260926 22:40:51.539302 20966 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:51.539449 20974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:51.539983 20974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:51.541661 20974 catalog_manager.cc:1347] Generated new cluster ID: 6066af668f9a443184783e2b08a3d13d
I20260926 22:40:51.541714 20974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:51.543898 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:51.537919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.11.107.254:36311)
0926 22:40:51.538153 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:51.538163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:51.538257 (+ 94us) server_negotiation.cc:407] Connection header received
0926 22:40:51.538384 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:51.538392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:51.538492 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:51.538928 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:51.539069 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.540629 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.541806 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.542423 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.542535 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:51.543223 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:51.543227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:51.543227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:51.543430 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:51.543469 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:51.543482 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:51.543541 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:51.543634 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 22:40:51.543708 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260926 22:40:51.562605 20974 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:51.563359 20974 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:51.574653 20974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d34bc9495b374e8a88b9df200169f51f: Generated new TSK 0
I20260926 22:40:51.574893 20974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:40:51.580665 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:36311
--builtin_ntp_servers=127.11.107.212:44689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 22:40:51.661706 20984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:51.661989 20984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:51.662014 20984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:51.663165 20984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:51.663204 20984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:51.663228 20984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:51.664518 20984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:44689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:36311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:51.664755 20984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:51.665035 20984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:51.670008 20990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:51.670042 20989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:51.670022 20992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:51.670038 20984 server_base.cc:1034] running on GCE node
I20260926 22:40:51.670411 20984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:51.670629 20984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:51.671824 20984 hybrid_clock.cc:648] HybridClock initialized: now 1790462451671810 us; error 23 us; skew 500 ppm
I20260926 22:40:51.673638 20984 webserver.cc:460] Webserver started at http://127.11.107.193:44305/ using document root <none> and password file <none>
I20260926 22:40:51.673979 20984 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:51.674042 20984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:51.674203 20984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:51.675832 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "0acf21363eaf4d8a8cc03b3a81a5948d"
format_stamp: "Formatted at 2026-09-26 22:40:51 on dist-test-slave-j8sq"
server_key: "20d96416ce10cfeb5e56c59775966871"
server_key_iv: "c9900e85a43b8a7c31338d66043880c5"
server_key_version: "encryptionkey@0"
I20260926 22:40:51.676229 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "0acf21363eaf4d8a8cc03b3a81a5948d"
format_stamp: "Formatted at 2026-09-26 22:40:51 on dist-test-slave-j8sq"
server_key: "20d96416ce10cfeb5e56c59775966871"
server_key_iv: "c9900e85a43b8a7c31338d66043880c5"
server_key_version: "encryptionkey@0"
I20260926 22:40:51.677824 20984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260926 22:40:51.678755 20998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:51.678937 20984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:51.679049 20984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "0acf21363eaf4d8a8cc03b3a81a5948d"
format_stamp: "Formatted at 2026-09-26 22:40:51 on dist-test-slave-j8sq"
server_key: "20d96416ce10cfeb5e56c59775966871"
server_key_iv: "c9900e85a43b8a7c31338d66043880c5"
server_key_version: "encryptionkey@0"
I20260926 22:40:51.679128 20984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:51.688028 20984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:51.688328 20984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:51.688470 20984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:51.688688 20984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:51.688997 20984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:51.689046 20984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:51.689086 20984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:51.689111 20984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:51.694849 20984 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:37051
I20260926 22:40:51.694891 21111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:37051 every 8 connection(s)
I20260926 22:40:51.695201 20984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:51.695693 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 20984
I20260926 22:40:51.695791 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:51.696014 11695 external_mini_cluster.cc:1428] Setting key 0af34e3ce43ae5c1747cefbd5fbc425b
I20260926 22:40:51.697044 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:36311
--builtin_ntp_servers=127.11.107.212:44689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 22:40:51.701316 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:51.695909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:38621 (local address 127.11.107.254:36311)
0926 22:40:51.695973 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:51.695975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:51.696374 (+ 399us) server_negotiation.cc:407] Connection header received
0926 22:40:51.696947 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:51.696948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:51.696962 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:51.697031 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:51.697798 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.698452 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.699712 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.700028 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.700101 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:51.701006 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:51.701007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:51.701008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:51.701012 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:51.701029 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:51.701034 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:51.701093 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:51.701171 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 22:40:51.701202 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:40:51.701403 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:51.695905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:36311 (local address 127.11.107.193:38621)
0926 22:40:51.696117 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:51.696162 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:51.696848 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:51.697084 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:51.697088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:51.697119 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:51.697708 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:51.697713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.698565 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.698566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:51.699623 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:51.699626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.700111 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.700112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:51.700263 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:51.700874 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:51.700874 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:51.700901 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:51.700922 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:51.701098 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:51.701102 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:51.701104 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:40:51.701154 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 22:40:51.701231 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260926 22:40:51.701989 21112 heartbeater.cc:347] Connected to a master server at 127.11.107.254:36311
I20260926 22:40:51.702085 21112 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:51.702306 21112 heartbeater.cc:511] Master 127.11.107.254:36311 requested a full tablet report, sending...
I20260926 22:40:51.702834 20924 ts_manager.cc:194] Registered new tserver with Master: 0acf21363eaf4d8a8cc03b3a81a5948d (127.11.107.193:37051)
I20260926 22:40:51.703471 20924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.107.193:38621
W20260926 22:40:51.776973 21115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:51.777186 21115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:51.777207 21115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:51.778363 21115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:51.778400 21115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:51.778424 21115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:51.779726 21115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:44689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:36311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:51.779974 21115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:51.780258 21115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:51.785089 21120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:51.785094 21123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:51.785068 21121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:51.785210 21115 server_base.cc:1034] running on GCE node
I20260926 22:40:51.785503 21115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:51.785751 21115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:51.786908 21115 hybrid_clock.cc:648] HybridClock initialized: now 1790462451786899 us; error 43 us; skew 500 ppm
I20260926 22:40:51.788036 21115 webserver.cc:460] Webserver started at http://127.11.107.194:42579/ using document root <none> and password file <none>
I20260926 22:40:51.788275 21115 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:51.788347 21115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:51.788487 21115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:51.789491 21115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "798d2fd939744214b4f3b3b93cf7dd40"
format_stamp: "Formatted at 2026-09-26 22:40:51 on dist-test-slave-j8sq"
server_key: "b36a1adb85ef3681fdfd102330b5150e"
server_key_iv: "49929ce6fa47d528fffd751a65524287"
server_key_version: "encryptionkey@0"
I20260926 22:40:51.789903 21115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "798d2fd939744214b4f3b3b93cf7dd40"
format_stamp: "Formatted at 2026-09-26 22:40:51 on dist-test-slave-j8sq"
server_key: "b36a1adb85ef3681fdfd102330b5150e"
server_key_iv: "49929ce6fa47d528fffd751a65524287"
server_key_version: "encryptionkey@0"
I20260926 22:40:51.791137 21115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:51.791880 21129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:51.792030 21115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:51.792128 21115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "798d2fd939744214b4f3b3b93cf7dd40"
format_stamp: "Formatted at 2026-09-26 22:40:51 on dist-test-slave-j8sq"
server_key: "b36a1adb85ef3681fdfd102330b5150e"
server_key_iv: "49929ce6fa47d528fffd751a65524287"
server_key_version: "encryptionkey@0"
I20260926 22:40:51.792212 21115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:51.800617 21115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:51.800875 21115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:51.801010 21115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:51.801226 21115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:51.801548 21115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:51.801622 21115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:51.801663 21115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:51.801690 21115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:51.807246 21115 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:38383
I20260926 22:40:51.807332 21242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:38383 every 8 connection(s)
I20260926 22:40:51.807611 21115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:40:51.812011 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 21115
I20260926 22:40:51.812114 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:51.812315 11695 external_mini_cluster.cc:1428] Setting key 994030f1afc51cabd7d73a091a9f3f24
I20260926 22:40:51.812959 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:51.808296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:47155 (local address 127.11.107.254:36311)
0926 22:40:51.808363 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:51.808365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:51.808578 (+ 213us) server_negotiation.cc:407] Connection header received
0926 22:40:51.809119 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:51.809121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:51.809134 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:51.809190 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:51.809911 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.810469 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.811457 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.811669 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.811752 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:51.812642 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:51.812644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:51.812644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:51.812649 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:51.812666 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:51.812671 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:51.812761 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:51.812853 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 22:40:51.812877 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 22:40:51.813187 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:51.808288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:36311 (local address 127.11.107.194:47155)
0926 22:40:51.808456 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:51.808485 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:51.808998 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:51.809256 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:51.809260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:51.809269 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:51.809780 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:51.809785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.810559 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.810560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:51.811357 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:51.811359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.811762 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.811763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:51.811890 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:51.812467 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:51.812468 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:51.812527 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:51.812556 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:51.812771 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:51.812776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:51.812779 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:51.812849 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 22:40:51.812947 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260926 22:40:51.813519 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:36311
--builtin_ntp_servers=127.11.107.212:44689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 22:40:51.813583 21243 heartbeater.cc:347] Connected to a master server at 127.11.107.254:36311
I20260926 22:40:51.813671 21243 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:51.813885 21243 heartbeater.cc:511] Master 127.11.107.254:36311 requested a full tablet report, sending...
I20260926 22:40:51.814229 20924 ts_manager.cc:194] Registered new tserver with Master: 798d2fd939744214b4f3b3b93cf7dd40 (127.11.107.194:38383)
I20260926 22:40:51.814594 20924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.107.194:47155
W20260926 22:40:51.896435 21246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:51.896641 21246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:51.896660 21246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:51.897782 21246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:51.897819 21246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:51.897867 21246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:51.899184 21246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:44689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:36311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:51.899374 21246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:51.899610 21246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:40:51.904645 21246 server_base.cc:1034] running on GCE node
W20260926 22:40:51.904636 21251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:51.904626 21254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:51.904618 21252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:51.905113 21246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:51.905383 21246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:51.906543 21246 hybrid_clock.cc:648] HybridClock initialized: now 1790462451906530 us; error 41 us; skew 500 ppm
I20260926 22:40:51.907716 21246 webserver.cc:460] Webserver started at http://127.11.107.195:45369/ using document root <none> and password file <none>
I20260926 22:40:51.907960 21246 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:51.908028 21246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:51.908165 21246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:51.909266 21246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "1b8f500d35a245ec9d7621fe78f0e23e"
format_stamp: "Formatted at 2026-09-26 22:40:51 on dist-test-slave-j8sq"
server_key: "78b830b58bfdf71cad84ad5315571bdd"
server_key_iv: "3b36bfe2a63a5e1c21c098a07c165b4a"
server_key_version: "encryptionkey@0"
I20260926 22:40:51.909631 21246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "1b8f500d35a245ec9d7621fe78f0e23e"
format_stamp: "Formatted at 2026-09-26 22:40:51 on dist-test-slave-j8sq"
server_key: "78b830b58bfdf71cad84ad5315571bdd"
server_key_iv: "3b36bfe2a63a5e1c21c098a07c165b4a"
server_key_version: "encryptionkey@0"
I20260926 22:40:51.911135 21246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:51.911845 21260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:51.912012 21246 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:51.912108 21246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "1b8f500d35a245ec9d7621fe78f0e23e"
format_stamp: "Formatted at 2026-09-26 22:40:51 on dist-test-slave-j8sq"
server_key: "78b830b58bfdf71cad84ad5315571bdd"
server_key_iv: "3b36bfe2a63a5e1c21c098a07c165b4a"
server_key_version: "encryptionkey@0"
I20260926 22:40:51.912194 21246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:51.921150 21246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:51.921450 21246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:51.921597 21246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:51.921806 21246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:51.922179 21246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:51.922237 21246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:51.922278 21246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:51.922304 21246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:51.927968 21246 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:42501
I20260926 22:40:51.928015 21373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:42501 every 8 connection(s)
I20260926 22:40:51.928351 21246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:40:51.929430 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 21246
I20260926 22:40:51.929533 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:51.929737 11695 external_mini_cluster.cc:1428] Setting key 52921a9fa1d7dd3687ae87793f7d31f7
I20260926 22:40:51.933717 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:51.929095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:44053 (local address 127.11.107.254:36311)
0926 22:40:51.929167 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:51.929170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:51.929510 (+ 340us) server_negotiation.cc:407] Connection header received
0926 22:40:51.930238 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:51.930239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:51.930252 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:51.930302 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:51.931107 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.931592 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.932527 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.932699 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.932749 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:51.933489 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:51.933490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:51.933490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:51.933494 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:51.933508 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:51.933512 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:51.933556 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:51.933639 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 22:40:51.933658 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 22:40:51.933921 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:51.929109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:36311 (local address 127.11.107.195:44053)
0926 22:40:51.929326 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:51.929375 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:51.930121 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:51.930392 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:51.930397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:51.930407 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:51.931022 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:51.931027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.931679 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.931679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:51.932453 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:51.932455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:51.932774 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:51.932775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:51.932879 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:51.933362 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:51.933363 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:51.933391 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:51.933411 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:51.933566 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:51.933569 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:51.933571 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:40:51.933622 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:40:51.933701 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260926 22:40:51.934414 21374 heartbeater.cc:347] Connected to a master server at 127.11.107.254:36311
I20260926 22:40:51.934511 21374 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:51.934686 21374 heartbeater.cc:511] Master 127.11.107.254:36311 requested a full tablet report, sending...
I20260926 22:40:51.935055 20924 ts_manager.cc:194] Registered new tserver with Master: 1b8f500d35a245ec9d7621fe78f0e23e (127.11.107.195:42501)
I20260926 22:40:51.935369 20924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.107.195:44053
I20260926 22:40:51.941248 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 22:40:51.943079 20964 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:51.942571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.11.107.254:36311)
0926 22:40:51.942626 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:51.942628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:51.942756 (+ 128us) server_negotiation.cc:407] Connection header received
0926 22:40:51.942826 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:51.942827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:51.942839 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:51.942892 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:51.943014 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53570: BlockingRecv error: recv got EOF from 127.0.0.1:53570 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 22:40:51.943135 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 20984
I20260926 22:40:51.949061 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 21115
I20260926 22:40:51.955420 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 21246
I20260926 22:40:51.959851 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 20893
2026-09-26T22:40:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (573 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T22:40:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:51Z Disabled control of system clock
I20260926 22:40:51.986011 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:38207
--webserver_interface=127.11.107.254
--webserver_port=0
--builtin_ntp_servers=127.11.107.212:40625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:38207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 22:40:52.067929 21385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:52.068138 21385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:52.068161 21385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:52.069198 21385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:52.069236 21385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:52.069252 21385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:52.069265 21385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:52.069278 21385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:52.070546 21385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:40625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:38207
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:38207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:52.070740 21385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:52.070979 21385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:52.075640 21393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:52.075628 21391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:52.075681 21385 server_base.cc:1034] running on GCE node
W20260926 22:40:52.075628 21390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:52.076141 21385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:52.076383 21385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:52.077523 21385 hybrid_clock.cc:648] HybridClock initialized: now 1790462452077510 us; error 29 us; skew 500 ppm
I20260926 22:40:52.078717 21385 webserver.cc:460] Webserver started at http://127.11.107.254:39513/ using document root <none> and password file <none>
I20260926 22:40:52.078902 21385 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:52.078941 21385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:52.079028 21385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:52.080067 21385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "16395214d0c94f0aa6aa33d5aa10c86b"
format_stamp: "Formatted at 2026-09-26 22:40:52 on dist-test-slave-j8sq"
server_key: "967e4a5537d9f4ae6ccc19e7c833bb8e"
server_key_iv: "54aa801e8e54fc2233783b64d24d9fa8"
server_key_version: "encryptionkey@0"
I20260926 22:40:52.080358 21385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "16395214d0c94f0aa6aa33d5aa10c86b"
format_stamp: "Formatted at 2026-09-26 22:40:52 on dist-test-slave-j8sq"
server_key: "967e4a5537d9f4ae6ccc19e7c833bb8e"
server_key_iv: "54aa801e8e54fc2233783b64d24d9fa8"
server_key_version: "encryptionkey@0"
I20260926 22:40:52.081533 21385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:40:52.082260 21399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:52.082432 21385 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:52.082515 21385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "16395214d0c94f0aa6aa33d5aa10c86b"
format_stamp: "Formatted at 2026-09-26 22:40:52 on dist-test-slave-j8sq"
server_key: "967e4a5537d9f4ae6ccc19e7c833bb8e"
server_key_iv: "54aa801e8e54fc2233783b64d24d9fa8"
server_key_version: "encryptionkey@0"
I20260926 22:40:52.082567 21385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:52.085680 21385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:52.085944 21385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:52.086045 21385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:52.089685 21385 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:38207
I20260926 22:40:52.089706 21451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:38207 every 8 connection(s)
I20260926 22:40:52.090082 21385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:52.090754 21452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:52.093411 21452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b: Bootstrap starting.
I20260926 22:40:52.094041 21452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:52.094331 21452 log.cc:824] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:52.094964 21452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b: No bootstrap required, opened a new log
I20260926 22:40:52.096470 21452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16395214d0c94f0aa6aa33d5aa10c86b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38207 } }
I20260926 22:40:52.096626 21452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:52.096709 21452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16395214d0c94f0aa6aa33d5aa10c86b, State: Initialized, Role: FOLLOWER
I20260926 22:40:52.096892 21452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16395214d0c94f0aa6aa33d5aa10c86b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38207 } }
I20260926 22:40:52.096989 21452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:52.097033 21452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:52.097078 21452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:52.097590 21452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16395214d0c94f0aa6aa33d5aa10c86b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38207 } }
I20260926 22:40:52.097710 21452 leader_election.cc:302] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16395214d0c94f0aa6aa33d5aa10c86b; no voters:
I20260926 22:40:52.097898 21452 leader_election.cc:288] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:52.097941 21455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:52.098097 21455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [term 1 LEADER]: Becoming Leader. State: Replica: 16395214d0c94f0aa6aa33d5aa10c86b, State: Running, Role: LEADER
I20260926 22:40:52.098153 21452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:40:52.098255 21455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16395214d0c94f0aa6aa33d5aa10c86b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38207 } }
I20260926 22:40:52.098641 21457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16395214d0c94f0aa6aa33d5aa10c86b. Latest consensus state: current_term: 1 leader_uuid: "16395214d0c94f0aa6aa33d5aa10c86b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16395214d0c94f0aa6aa33d5aa10c86b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38207 } } }
I20260926 22:40:52.098800 21457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:52.098845 21456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16395214d0c94f0aa6aa33d5aa10c86b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16395214d0c94f0aa6aa33d5aa10c86b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 38207 } } }
I20260926 22:40:52.098941 21456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:52.099285 21466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:52.099751 21466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:52.099818 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 21385
I20260926 22:40:52.099915 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:52.100103 11695 external_mini_cluster.cc:1428] Setting key bc54607f1df3de8446e633cde21991a4
I20260926 22:40:52.101475 21466 catalog_manager.cc:1347] Generated new cluster ID: 4a51cbf51fd442708f6c9345d7af3456
I20260926 22:40:52.101526 21466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:52.111342 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:52.100787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.11.107.254:38207)
0926 22:40:52.100968 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:52.100977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:52.101197 (+ 220us) server_negotiation.cc:407] Connection header received
0926 22:40:52.101343 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:52.101349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:52.101401 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:52.110039 (+ 8638us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:52.110870 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:52.110874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:52.110874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:52.110880 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:52.110925 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:52.110940 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:52.111005 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:52.111078 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 22:40:52.111134 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260926 22:40:52.125901 21466 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:52.126092 21466 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:52.137467 21466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16395214d0c94f0aa6aa33d5aa10c86b: Generated new TSK 0
I20260926 22:40:52.137681 21466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:40:52.142335 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38207
--builtin_ntp_servers=127.11.107.212:40625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 22:40:52.220996 21476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:52.221222 21476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:52.221258 21476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:52.222442 21476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:52.222502 21476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:52.222543 21476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:52.223843 21476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:40625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:52.224103 21476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:52.224385 21476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:52.229427 21484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:52.229466 21476 server_base.cc:1034] running on GCE node
W20260926 22:40:52.229446 21481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:52.229446 21482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:52.229938 21476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:52.230204 21476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:52.231364 21476 hybrid_clock.cc:648] HybridClock initialized: now 1790462452231354 us; error 43 us; skew 500 ppm
I20260926 22:40:52.232586 21476 webserver.cc:460] Webserver started at http://127.11.107.193:36153/ using document root <none> and password file <none>
I20260926 22:40:52.232821 21476 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:52.232889 21476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:52.233023 21476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:52.234113 21476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "ff2fb50d16314768b65c6c5ae1624570"
format_stamp: "Formatted at 2026-09-26 22:40:52 on dist-test-slave-j8sq"
server_key: "713653b640bcb937e4ea45ed510ec50b"
server_key_iv: "559e670a2ef7e8870bc918c559be5684"
server_key_version: "encryptionkey@0"
I20260926 22:40:52.234476 21476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "ff2fb50d16314768b65c6c5ae1624570"
format_stamp: "Formatted at 2026-09-26 22:40:52 on dist-test-slave-j8sq"
server_key: "713653b640bcb937e4ea45ed510ec50b"
server_key_iv: "559e670a2ef7e8870bc918c559be5684"
server_key_version: "encryptionkey@0"
I20260926 22:40:52.235704 21476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:52.236363 21490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:52.236542 21476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:52.236639 21476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "ff2fb50d16314768b65c6c5ae1624570"
format_stamp: "Formatted at 2026-09-26 22:40:52 on dist-test-slave-j8sq"
server_key: "713653b640bcb937e4ea45ed510ec50b"
server_key_iv: "559e670a2ef7e8870bc918c559be5684"
server_key_version: "encryptionkey@0"
I20260926 22:40:52.236696 21476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:52.239746 21476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:52.239996 21476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:52.240134 21476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:52.240353 21476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:52.240631 21476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:52.240686 21476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:52.240725 21476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:52.240751 21476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:52.246310 21476 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:37145
I20260926 22:40:52.246376 21603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:37145 every 8 connection(s)
I20260926 22:40:52.246711 21476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:52.246836 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 21476
I20260926 22:40:52.246935 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:52.247133 11695 external_mini_cluster.cc:1428] Setting key 5b1c799c6a96931dcec06fc77b24ef21
I20260926 22:40:52.248135 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38207
--builtin_ntp_servers=127.11.107.212:40625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 22:40:52.249889 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:52.247474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:57367 (local address 127.11.107.254:38207)
0926 22:40:52.247515 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:52.247517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:52.247755 (+ 238us) server_negotiation.cc:407] Connection header received
0926 22:40:52.248309 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:52.248311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:52.248320 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:52.248349 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:52.249584 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:52.249585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:52.249586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:52.249591 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:52.249606 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:52.249611 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:52.249677 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:52.249768 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 22:40:52.249785 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:40:52.250056 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:52.247467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:38207 (local address 127.11.107.193:57367)
0926 22:40:52.247654 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:52.247683 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:52.248167 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:52.248499 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:52.248502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:52.248509 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:52.249439 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:52.249439 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:52.249464 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:52.249490 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:52.249678 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:52.249683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:52.249686 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:52.249777 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 22:40:52.249833 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260926 22:40:52.250519 21604 heartbeater.cc:347] Connected to a master server at 127.11.107.254:38207
I20260926 22:40:52.250602 21604 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:52.250756 21604 heartbeater.cc:511] Master 127.11.107.254:38207 requested a full tablet report, sending...
I20260926 22:40:52.251294 21416 ts_manager.cc:194] Registered new tserver with Master: ff2fb50d16314768b65c6c5ae1624570 (127.11.107.193:37145)
W20260926 22:40:52.327504 21607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:52.327694 21607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:52.327716 21607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:52.328773 21607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:52.328804 21607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:52.328826 21607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:52.330143 21607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:40625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:52.330325 21607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:52.330559 21607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:52.335248 21612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:52.335238 21613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:52.335239 21615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:52.335421 21607 server_base.cc:1034] running on GCE node
I20260926 22:40:52.335739 21607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:52.335973 21607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:52.337142 21607 hybrid_clock.cc:648] HybridClock initialized: now 1790462452337142 us; error 46 us; skew 500 ppm
I20260926 22:40:52.338346 21607 webserver.cc:460] Webserver started at http://127.11.107.194:36411/ using document root <none> and password file <none>
I20260926 22:40:52.338580 21607 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:52.338662 21607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:52.338851 21607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:52.339829 21607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "e4d096b77fbf49729386808f4892c323"
format_stamp: "Formatted at 2026-09-26 22:40:52 on dist-test-slave-j8sq"
server_key: "f43c638d2ef599be74e74a626025cdbb"
server_key_iv: "729a7dc74964c5b68bbd9e64121a63a9"
server_key_version: "encryptionkey@0"
I20260926 22:40:52.340202 21607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "e4d096b77fbf49729386808f4892c323"
format_stamp: "Formatted at 2026-09-26 22:40:52 on dist-test-slave-j8sq"
server_key: "f43c638d2ef599be74e74a626025cdbb"
server_key_iv: "729a7dc74964c5b68bbd9e64121a63a9"
server_key_version: "encryptionkey@0"
I20260926 22:40:52.341579 21607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:40:52.342337 21621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:52.342499 21607 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:52.342589 21607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "e4d096b77fbf49729386808f4892c323"
format_stamp: "Formatted at 2026-09-26 22:40:52 on dist-test-slave-j8sq"
server_key: "f43c638d2ef599be74e74a626025cdbb"
server_key_iv: "729a7dc74964c5b68bbd9e64121a63a9"
server_key_version: "encryptionkey@0"
I20260926 22:40:52.342693 21607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:52.345793 21607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:52.346051 21607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:52.346184 21607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:52.346393 21607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:52.346680 21607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:52.346732 21607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:52.346769 21607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:52.346796 21607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:52.352103 21607 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:35041
I20260926 22:40:52.352185 21734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:35041 every 8 connection(s)
I20260926 22:40:52.352474 21607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:40:52.352896 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 21607
I20260926 22:40:52.353008 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:52.353152 11695 external_mini_cluster.cc:1428] Setting key de1649a704dfb3945ecd60484a0fe791
I20260926 22:40:52.354158 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38207
--builtin_ntp_servers=127.11.107.212:40625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 22:40:52.356199 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:52.353691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:48751 (local address 127.11.107.254:38207)
0926 22:40:52.353734 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:52.353736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:52.354017 (+ 281us) server_negotiation.cc:407] Connection header received
0926 22:40:52.354688 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:52.354688 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:52.354694 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:52.354732 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:52.355954 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:52.355955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:52.355956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:52.355960 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:52.355981 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:52.355986 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:52.356058 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:52.356105 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 22:40:52.356120 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:40:52.356362 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:52.353694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:38207 (local address 127.11.107.194:48751)
0926 22:40:52.353903 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:52.353946 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:52.354580 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:52.354808 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:52.354811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:52.354827 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:52.355809 (+ 982us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:52.355809 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:52.355835 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:52.355862 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:52.356055 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:52.356057 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:52.356059 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:40:52.356107 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 22:40:52.356151 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260926 22:40:52.356873 21735 heartbeater.cc:347] Connected to a master server at 127.11.107.254:38207
I20260926 22:40:52.356967 21735 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:52.357115 21735 heartbeater.cc:511] Master 127.11.107.254:38207 requested a full tablet report, sending...
I20260926 22:40:52.357465 21416 ts_manager.cc:194] Registered new tserver with Master: e4d096b77fbf49729386808f4892c323 (127.11.107.194:35041)
W20260926 22:40:52.434140 21738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:52.434327 21738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:52.434350 21738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:52.435459 21738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:52.435493 21738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:52.435518 21738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:52.436776 21738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:40625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:38207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:52.436954 21738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:52.437199 21738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:52.442124 21743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:52.442129 21744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:52.442142 21746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:52.442184 21738 server_base.cc:1034] running on GCE node
I20260926 22:40:52.442577 21738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:52.442828 21738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:52.443979 21738 hybrid_clock.cc:648] HybridClock initialized: now 1790462452443960 us; error 37 us; skew 500 ppm
I20260926 22:40:52.445246 21738 webserver.cc:460] Webserver started at http://127.11.107.195:42233/ using document root <none> and password file <none>
I20260926 22:40:52.445490 21738 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:52.445537 21738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:52.445712 21738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:52.446825 21738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "b827d2789f3b43b6995d155a0552f2d4"
format_stamp: "Formatted at 2026-09-26 22:40:52 on dist-test-slave-j8sq"
server_key: "2bbdc5005285412d06c5a9208c410caa"
server_key_iv: "3c895c75b0126f866a2ab4f669b9af15"
server_key_version: "encryptionkey@0"
I20260926 22:40:52.447180 21738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "b827d2789f3b43b6995d155a0552f2d4"
format_stamp: "Formatted at 2026-09-26 22:40:52 on dist-test-slave-j8sq"
server_key: "2bbdc5005285412d06c5a9208c410caa"
server_key_iv: "3c895c75b0126f866a2ab4f669b9af15"
server_key_version: "encryptionkey@0"
I20260926 22:40:52.448421 21738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:52.449249 21752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:52.449446 21738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:40:52.449544 21738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "b827d2789f3b43b6995d155a0552f2d4"
format_stamp: "Formatted at 2026-09-26 22:40:52 on dist-test-slave-j8sq"
server_key: "2bbdc5005285412d06c5a9208c410caa"
server_key_iv: "3c895c75b0126f866a2ab4f669b9af15"
server_key_version: "encryptionkey@0"
I20260926 22:40:52.449632 21738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:52.452868 21738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:52.453119 21738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:52.453263 21738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:52.453470 21738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:52.453750 21738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:52.453804 21738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:52.453857 21738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:52.453881 21738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:52.459486 21738 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:33975
I20260926 22:40:52.459539 21865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:33975 every 8 connection(s)
I20260926 22:40:52.459836 21738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:40:52.462643 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:52.460509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:37259 (local address 127.11.107.254:38207)
0926 22:40:52.460589 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:52.460591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:52.460776 (+ 185us) server_negotiation.cc:407] Connection header received
0926 22:40:52.461229 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:52.461231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:52.461237 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:52.461309 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:52.462438 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:52.462439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:52.462439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:40:52.462443 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:52.462456 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:40:52.462461 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:52.462518 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:52.462579 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 22:40:52.462592 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 22:40:52.462821 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:52.460474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:38207 (local address 127.11.107.195:37259)
0926 22:40:52.460665 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:52.460696 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:52.461113 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:52.461359 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:52.461362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:52.461370 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:52.462297 (+ 927us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:40:52.462298 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:52.462323 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:40:52.462351 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:52.462518 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:52.462520 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:52.462522 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:40:52.462571 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:40:52.462613 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260926 22:40:52.463282 21866 heartbeater.cc:347] Connected to a master server at 127.11.107.254:38207
I20260926 22:40:52.463378 21866 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:52.463515 21866 heartbeater.cc:511] Master 127.11.107.254:38207 requested a full tablet report, sending...
I20260926 22:40:52.463840 21416 ts_manager.cc:194] Registered new tserver with Master: b827d2789f3b43b6995d155a0552f2d4 (127.11.107.195:33975)
I20260926 22:40:52.469705 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 21738
I20260926 22:40:52.469786 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:52.469988 11695 external_mini_cluster.cc:1428] Setting key 0197ef2a78af6b072cef830aa66b2680
I20260926 22:40:52.470840 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 22:40:52.472687 21474 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:40:52.472129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.11.107.254:38207)
0926 22:40:52.472205 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:52.472207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:52.472333 (+ 126us) server_negotiation.cc:407] Connection header received
0926 22:40:52.472364 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:52.472365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:52.472372 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:52.472448 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:52.472621 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54420: BlockingRecv error: recv got EOF from 127.0.0.1:54420 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 22:40:52.472702 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 21476
I20260926 22:40:52.477161 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 21607
I20260926 22:40:52.481004 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 21738
I20260926 22:40:52.485100 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 21385
2026-09-26T22:40:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (524 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:52 dist-test-slave-j8sq krb5kdc[21876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:52 dist-test-slave-j8sq krb5kdc[21876](info): set up 2 sockets
Sep 26 22:40:52 dist-test-slave-j8sq krb5kdc[21876](info): commencing operation
krb5kdc: starting...
W20260926 22:40:53.507639 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:53 dist-test-slave-j8sq krb5kdc[21876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:40:53.624408 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:39955
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:39955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:53.706789 21889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:53.707015 21889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:53.707051 21889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:53.708166 21889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:40:53.708238 21889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:53.708262 21889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:53.708283 21889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:40:53.708312 21889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:40:53.709722 21889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:39955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:39955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:53.710024 21889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:53.710297 21889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:53.715153 21897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:53.715194 21894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:53.715168 21895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:53.715219 21889 server_base.cc:1034] running on GCE node
I20260926 22:40:53.715596 21889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:53.715891 21889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:53.717032 21889 hybrid_clock.cc:648] HybridClock initialized: now 1790462453717019 us; error 28 us; skew 500 ppm
Sep 26 22:40:53 dist-test-slave-j8sq krb5kdc[21876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462453, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:53.719530 21889 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:40:53.719998 21889 webserver.cc:460] Webserver started at http://127.11.107.254:39007/ using document root <none> and password file <none>
I20260926 22:40:53.720234 21889 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:53.720301 21889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:53.720448 21889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:53.721490 21889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "82013e4c3ad94f1686d00508b22e0a63"
format_stamp: "Formatted at 2026-09-26 22:40:53 on dist-test-slave-j8sq"
server_key: "ff874b642eb791baf2279e88e3e686a2"
server_key_iv: "b6ad398294e6a0d09cb4f45ed1dd6ce0"
server_key_version: "encryptionkey@0"
I20260926 22:40:53.721912 21889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "82013e4c3ad94f1686d00508b22e0a63"
format_stamp: "Formatted at 2026-09-26 22:40:53 on dist-test-slave-j8sq"
server_key: "ff874b642eb791baf2279e88e3e686a2"
server_key_iv: "b6ad398294e6a0d09cb4f45ed1dd6ce0"
server_key_version: "encryptionkey@0"
I20260926 22:40:53.723227 21889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:40:53.723908 21904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:53.724073 21889 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:53.724169 21889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "82013e4c3ad94f1686d00508b22e0a63"
format_stamp: "Formatted at 2026-09-26 22:40:53 on dist-test-slave-j8sq"
server_key: "ff874b642eb791baf2279e88e3e686a2"
server_key_iv: "b6ad398294e6a0d09cb4f45ed1dd6ce0"
server_key_version: "encryptionkey@0"
I20260926 22:40:53.724253 21889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:53.749482 21889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:53.751786 21889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:53.751946 21889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:53.756080 21889 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:39955
I20260926 22:40:53.756115 21956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:39955 every 8 connection(s)
I20260926 22:40:53.756464 21889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:40:53.757416 21957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:53.759099 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 21889
I20260926 22:40:53.759193 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:40:53.759397 11695 external_mini_cluster.cc:1428] Setting key d5ad614e049dbb90d80db4a2c9ccac88
I20260926 22:40:53.760140 21957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63: Bootstrap starting.
I20260926 22:40:53.760694 21957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:53.760996 21957 log.cc:824] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:53.761720 21957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63: No bootstrap required, opened a new log
I20260926 22:40:53.763459 21957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013e4c3ad94f1686d00508b22e0a63" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39955 } }
I20260926 22:40:53.763617 21957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:53.763664 21957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82013e4c3ad94f1686d00508b22e0a63, State: Initialized, Role: FOLLOWER
I20260926 22:40:53.763831 21957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013e4c3ad94f1686d00508b22e0a63" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39955 } }
I20260926 22:40:53.763942 21957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:40:53.763979 21957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:40:53.764034 21957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:53.764796 21957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013e4c3ad94f1686d00508b22e0a63" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39955 } }
I20260926 22:40:53.764931 21957 leader_election.cc:302] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82013e4c3ad94f1686d00508b22e0a63; no voters:
I20260926 22:40:53.765089 21957 leader_election.cc:288] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:40:53.765154 21962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:53.765372 21962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [term 1 LEADER]: Becoming Leader. State: Replica: 82013e4c3ad94f1686d00508b22e0a63, State: Running, Role: LEADER
I20260926 22:40:53.765355 21957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 22:40:53 dist-test-slave-j8sq krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:53.766577 21962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013e4c3ad94f1686d00508b22e0a63" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39955 } }
I20260926 22:40:53.767169 21964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82013e4c3ad94f1686d00508b22e0a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013e4c3ad94f1686d00508b22e0a63" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39955 } } }
I20260926 22:40:53.767266 21964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [sys.catalog]: This master's current role is: LEADER
I20260926 22:40:53.767165 21963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82013e4c3ad94f1686d00508b22e0a63. Latest consensus state: current_term: 1 leader_uuid: "82013e4c3ad94f1686d00508b22e0a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013e4c3ad94f1686d00508b22e0a63" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39955 } } }
I20260926 22:40:53.767405 21963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63 [sys.catalog]: This master's current role is: LEADER
W20260926 22:40:53.768008 21976 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:40:53.768607 21976 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:40:53.768735 21978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:40:53.768931 21978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:40:53.770167 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:53.759912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.11.107.254:39955)
0926 22:40:53.760093 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:53.760101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:53.760125 (+ 24us) server_negotiation.cc:407] Connection header received
0926 22:40:53.760587 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:53.760594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:53.760639 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:53.760791 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:53.761509 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:53.762469 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:53.763232 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:53.763629 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:53.763751 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:53.766533 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:53.766537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:53.766537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:53.766548 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:53.768408 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:53.768859 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:53.768862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:53.768862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:53.768962 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:53.769176 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:53.769177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:53.769177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:53.769472 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:53.769627 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:53.769733 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 22:40:53.769964 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260926 22:40:53.770533 21978 catalog_manager.cc:1347] Generated new cluster ID: 45dc1cc0246c4566a4ec0fd74e4345a3
I20260926 22:40:53.770579 21978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:40:53.780833 21978 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:40:53.781518 21978 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:40:53.787565 21978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63: Generated new TSK 0
I20260926 22:40:53.787729 21978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:40:53.816049 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:39955
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:53.896111 21983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:53.896308 21983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:53.896330 21983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:53.897436 21983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:53.897475 21983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:53.897526 21983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:40:53.898981 21983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:53.899175 21983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:53.899379 21983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:53.904204 21988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:53.904299 21983 server_base.cc:1034] running on GCE node
W20260926 22:40:53.904210 21989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:53.904362 21991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:53.904640 21983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:53.904906 21983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:53.906040 21983 hybrid_clock.cc:648] HybridClock initialized: now 1790462453906020 us; error 35 us; skew 500 ppm
Sep 26 22:40:53 dist-test-slave-j8sq krb5kdc[21876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462453, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:53.908361 21983 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:40:53.908785 21983 webserver.cc:460] Webserver started at http://127.11.107.193:34371/ using document root <none> and password file <none>
I20260926 22:40:53.909004 21983 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:53.909046 21983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:53.909193 21983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:53.910300 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "6565dd061c244bd8a6bfefffd271233e"
format_stamp: "Formatted at 2026-09-26 22:40:53 on dist-test-slave-j8sq"
server_key: "0490c72bd7265a511848915c24ede47d"
server_key_iv: "92687692b0a516b6230e27765d3bd2c6"
server_key_version: "encryptionkey@0"
I20260926 22:40:53.910652 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "6565dd061c244bd8a6bfefffd271233e"
format_stamp: "Formatted at 2026-09-26 22:40:53 on dist-test-slave-j8sq"
server_key: "0490c72bd7265a511848915c24ede47d"
server_key_iv: "92687692b0a516b6230e27765d3bd2c6"
server_key_version: "encryptionkey@0"
I20260926 22:40:53.911949 21983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:53.912659 21998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:53.912845 21983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:53.912920 21983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "6565dd061c244bd8a6bfefffd271233e"
format_stamp: "Formatted at 2026-09-26 22:40:53 on dist-test-slave-j8sq"
server_key: "0490c72bd7265a511848915c24ede47d"
server_key_iv: "92687692b0a516b6230e27765d3bd2c6"
server_key_version: "encryptionkey@0"
I20260926 22:40:53.912972 21983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:53.926362 21983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:53.928503 21983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:53.928615 21983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:53.928797 21983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:53.929117 21983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:53.929185 21983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:53.929229 21983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:53.929260 21983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:53.934675 21983 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:38915
I20260926 22:40:53.934746 22111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:38915 every 8 connection(s)
I20260926 22:40:53.935070 21983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:40:53.940375 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 21983
I20260926 22:40:53.940451 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:40:53.940631 11695 external_mini_cluster.cc:1428] Setting key 2ebaed01fd0c707b3262bb760ec7ce57
Sep 26 22:40:53 dist-test-slave-j8sq krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462453, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:53.943987 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:53.935806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:39955 (local address 127.11.107.193:46593)
0926 22:40:53.935953 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:53.935985 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:53.936432 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:53.936641 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:53.936644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:53.937181 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:53.937360 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:53.937364 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:53.938001 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:53.938002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:53.938791 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:53.938792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:53.939123 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:53.939124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:53.939215 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:53.939713 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:53.939721 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:53.941083 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:53.942689 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:53.942695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:53.942696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:53.942909 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:53.943117 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:53.943118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:53.943119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:53.943227 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:53.943517 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:53.943522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:53.943640 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 22:40:53.943731 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 22:40:53.943841 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260926 22:40:53.944006 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:53.935824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:46593 (local address 127.11.107.254:39955)
0926 22:40:53.935867 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:53.935869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:53.936070 (+ 201us) server_negotiation.cc:407] Connection header received
0926 22:40:53.936508 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:53.936509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:53.936519 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:53.936569 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:53.937431 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:53.937931 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:53.938861 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:53.939050 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:53.939098 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:53.941187 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:53.941189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:53.941189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:53.941196 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:53.942577 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:53.942987 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:53.942988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:53.942989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:53.943049 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:53.943306 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:53.943307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:53.943307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:53.943445 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:53.943508 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:53.943776 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 22:40:53.943865 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:40:53.944454 22112 heartbeater.cc:347] Connected to a master server at 127.11.107.254:39955
I20260926 22:40:53.944550 22112 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:53.944734 22112 heartbeater.cc:511] Master 127.11.107.254:39955 requested a full tablet report, sending...
I20260926 22:40:53.945281 21921 ts_manager.cc:194] Registered new tserver with Master: 6565dd061c244bd8a6bfefffd271233e (127.11.107.193:38915)
I20260926 22:40:53.946023 21921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:46593
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:40:53.965080 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:39955
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:54.044188 22117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:54.044414 22117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:54.044452 22117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:54.045739 22117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:54.045804 22117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:54.045849 22117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:40:54.047379 22117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:54.047559 22117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:54.047739 22117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:54.052631 22125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:54.052655 22122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:54.052631 22123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:54.052906 22117 server_base.cc:1034] running on GCE node
I20260926 22:40:54.053098 22117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:54.053342 22117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:54.054486 22117 hybrid_clock.cc:648] HybridClock initialized: now 1790462454054478 us; error 32 us; skew 500 ppm
Sep 26 22:40:54 dist-test-slave-j8sq krb5kdc[21876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462454, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:54.056860 22117 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:40:54.057294 22117 webserver.cc:460] Webserver started at http://127.11.107.194:46091/ using document root <none> and password file <none>
I20260926 22:40:54.057487 22117 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:54.057526 22117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:54.057610 22117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:54.058661 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "ab418466dea54211924b9afe428df3bf"
format_stamp: "Formatted at 2026-09-26 22:40:54 on dist-test-slave-j8sq"
server_key: "ba474725482fa4c2ea8917e610b15fd1"
server_key_iv: "4efcff33b8af2da36b23f82dd21557f9"
server_key_version: "encryptionkey@0"
I20260926 22:40:54.058967 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "ab418466dea54211924b9afe428df3bf"
format_stamp: "Formatted at 2026-09-26 22:40:54 on dist-test-slave-j8sq"
server_key: "ba474725482fa4c2ea8917e610b15fd1"
server_key_iv: "4efcff33b8af2da36b23f82dd21557f9"
server_key_version: "encryptionkey@0"
I20260926 22:40:54.060168 22117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:40:54.060825 22132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:54.060979 22117 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:40:54.061093 22117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "ab418466dea54211924b9afe428df3bf"
format_stamp: "Formatted at 2026-09-26 22:40:54 on dist-test-slave-j8sq"
server_key: "ba474725482fa4c2ea8917e610b15fd1"
server_key_iv: "4efcff33b8af2da36b23f82dd21557f9"
server_key_version: "encryptionkey@0"
I20260926 22:40:54.061172 22117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:54.070612 22117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:54.072306 22117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:54.072459 22117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:54.072682 22117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:54.072979 22117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:54.073032 22117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:54.073073 22117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:54.073100 22117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:54.078975 22117 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:39619
I20260926 22:40:54.079042 22245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:39619 every 8 connection(s)
I20260926 22:40:54.079384 22117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:40:54 dist-test-slave-j8sq krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462454, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:54.088378 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.080137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:54689 (local address 127.11.107.254:39955)
0926 22:40:54.080188 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:54.080190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:54.080408 (+ 218us) server_negotiation.cc:407] Connection header received
0926 22:40:54.080930 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:54.080932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:54.080943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:54.081008 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:54.081951 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.082459 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.083459 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.083664 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.083752 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.085818 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:54.085820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:54.085820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:54.085828 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:54.087019 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:54.087414 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:54.087415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:54.087416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:54.087474 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:54.087716 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:54.087717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:54.087718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:54.087851 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:54.087923 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:54.088195 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 22:40:54.088288 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:40:54.088546 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.080110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:39955 (local address 127.11.107.194:54689)
0926 22:40:54.080289 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:54.080322 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:54.080826 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:54.081076 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:54.081080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:54.081634 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:54.081825 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.081830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.082552 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.082553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.083364 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.083366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.083760 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.083761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.083890 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.084377 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:54.084385 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:54.085720 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:54.087111 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:54.087118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:54.087120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:54.087327 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:54.087548 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:54.087548 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:54.087549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:54.087646 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:54.087923 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:54.087931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:54.088061 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 22:40:54.088167 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 22:40:54.088316 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260926 22:40:54.089062 22246 heartbeater.cc:347] Connected to a master server at 127.11.107.254:39955
I20260926 22:40:54.089139 22246 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:54.089285 22246 heartbeater.cc:511] Master 127.11.107.254:39955 requested a full tablet report, sending...
I20260926 22:40:54.089565 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 22117
I20260926 22:40:54.089672 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:40:54.089753 21921 ts_manager.cc:194] Registered new tserver with Master: ab418466dea54211924b9afe428df3bf (127.11.107.194:39619)
I20260926 22:40:54.089942 11695 external_mini_cluster.cc:1428] Setting key 906d6d0f62058ee8c0a33dcc3a9b75fb
I20260926 22:40:54.090240 21921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:54689
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:40:54.112751 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:39955
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:40:54.192006 22251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:40:54.192234 22251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:40:54.192288 22251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:40:54.193436 22251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:40:54.193503 22251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:40:54.193547 22251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:40:54.195011 22251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:40:54.195274 22251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:40:54.195490 22251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:40:54.200292 22257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:40:54.200352 22259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:54.200389 22251 server_base.cc:1034] running on GCE node
W20260926 22:40:54.200367 22256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:40:54.200736 22251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:40:54.200984 22251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:40:54.202137 22251 hybrid_clock.cc:648] HybridClock initialized: now 1790462454202148 us; error 58 us; skew 500 ppm
Sep 26 22:40:54 dist-test-slave-j8sq krb5kdc[21876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462454, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:40:54.204660 22251 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:40:54.205035 22251 webserver.cc:460] Webserver started at http://127.11.107.195:37081/ using document root <none> and password file <none>
I20260926 22:40:54.205214 22251 fs_manager.cc:359] Metadata directory not provided
I20260926 22:40:54.205255 22251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:40:54.205341 22251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:40:54.206362 22251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "d2ebc12ddc984a5db173fa5163c7c7cd"
format_stamp: "Formatted at 2026-09-26 22:40:54 on dist-test-slave-j8sq"
server_key: "5cf6c242344674759cab7f7e2058233a"
server_key_iv: "afde959ad56d2689a1ef4d362abb108f"
server_key_version: "encryptionkey@0"
I20260926 22:40:54.206693 22251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "d2ebc12ddc984a5db173fa5163c7c7cd"
format_stamp: "Formatted at 2026-09-26 22:40:54 on dist-test-slave-j8sq"
server_key: "5cf6c242344674759cab7f7e2058233a"
server_key_iv: "afde959ad56d2689a1ef4d362abb108f"
server_key_version: "encryptionkey@0"
I20260926 22:40:54.207918 22251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:40:54.208539 22266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:54.208720 22251 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:40:54.208801 22251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "d2ebc12ddc984a5db173fa5163c7c7cd"
format_stamp: "Formatted at 2026-09-26 22:40:54 on dist-test-slave-j8sq"
server_key: "5cf6c242344674759cab7f7e2058233a"
server_key_iv: "afde959ad56d2689a1ef4d362abb108f"
server_key_version: "encryptionkey@0"
I20260926 22:40:54.208849 22251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:40:54.220129 22251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:40:54.221937 22251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:40:54.222047 22251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:40:54.222214 22251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:40:54.222519 22251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:40:54.222556 22251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:54.222611 22251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:40:54.222643 22251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:40:54.228608 22251 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:33887
I20260926 22:40:54.228683 22379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:33887 every 8 connection(s)
I20260926 22:40:54.228974 22251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:40:54 dist-test-slave-j8sq krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462454, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:40:54.237169 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 22251
I20260926 22:40:54.237303 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:40:54.237537 11695 external_mini_cluster.cc:1428] Setting key 76dce8681e6c5e5fb68155540a720910
I20260926 22:40:54.237911 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.229691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:39557 (local address 127.11.107.254:39955)
0926 22:40:54.229753 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:54.229755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:54.229980 (+ 225us) server_negotiation.cc:407] Connection header received
0926 22:40:54.230446 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:54.230447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:54.230455 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:54.230510 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:54.231361 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.231847 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.232785 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.232959 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.233012 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.235335 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:54.235336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:54.235336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:54.235342 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:54.236558 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:54.236920 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:54.236921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:54.236921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:54.236969 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:54.237214 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:54.237215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:54.237215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:54.237369 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:54.237443 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:54.237710 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 22:40:54.237776 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 22:40:54.238174 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.229672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:39955 (local address 127.11.107.195:39557)
0926 22:40:54.229855 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:54.229896 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:54.230364 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:54.230567 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:54.230570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:54.231105 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:40:54.231289 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.231293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.231931 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.231931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.232715 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.232716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.233059 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.233060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.233166 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.233717 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:40:54.233725 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:40:54.235248 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:40:54.236632 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:54.236637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:54.236639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:54.236857 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:54.237055 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:40:54.237056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:40:54.237056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:40:54.237148 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:40:54.237439 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:40:54.237444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:40:54.237561 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 22:40:54.237683 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 22:40:54.237935 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":102,"threads_started":1}
I20260926 22:40:54.238627 22380 heartbeater.cc:347] Connected to a master server at 127.11.107.254:39955
I20260926 22:40:54.238742 22380 heartbeater.cc:464] Registering TS with master...
I20260926 22:40:54.238920 22380 heartbeater.cc:511] Master 127.11.107.254:39955 requested a full tablet report, sending...
I20260926 22:40:54.239297 21920 ts_manager.cc:194] Registered new tserver with Master: d2ebc12ddc984a5db173fa5163c7c7cd (127.11.107.195:33887)
I20260926 22:40:54.239691 21920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:39557
I20260926 22:40:54.249459 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:40:54.257544 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.250880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.11.107.254:39955)
0926 22:40:54.250953 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:54.250956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:54.251051 (+ 95us) server_negotiation.cc:407] Connection header received
0926 22:40:54.251131 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:54.251132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:54.251141 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:54.251210 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:40:54.251791 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.252286 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.253122 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.253323 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.253400 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.254643 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:40:54.254644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:40:54.254645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:40:54.254652 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:40:54.256377 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:54.256736 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:54.256737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:54.256737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:54.256799 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:40:54.256994 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:40:54.256995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:40:54.256996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:40:54.257148 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:40:54.257261 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:54.257363 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 22:40:54.257443 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 22:40:54.259137 21920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:40:54.260651 21920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:40:54.267304 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.262847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:39619 (local address 127.0.0.1:55586)
0926 22:40:54.263134 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:54.263189 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:54.263263 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:54.263545 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:54.263563 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:54.263565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:54.263631 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.263633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.264772 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.264773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.266096 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.266099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.266989 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.266990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.267066 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.267080 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 22:40:54.267169 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 22:40:54.267223 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":78,"threads_started":1}
I20260926 22:40:54.267319 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.262964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.11.107.194:39619)
0926 22:40:54.263374 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:54.263379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:54.263396 (+ 17us) server_negotiation.cc:407] Connection header received
0926 22:40:54.263430 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:54.263431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:54.263443 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:54.263542 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:54.263735 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.264644 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.266197 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.266908 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.266983 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.267035 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:54.267193 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 22:40:54.267237 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":91,"threads_started":1}
I20260926 22:40:54.267531 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.263043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:38915 (local address 127.0.0.1:51178)
0926 22:40:54.263636 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:54.263656 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:54.263706 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:54.263896 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:54.263896 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:54.263898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:54.263943 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.263944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.265264 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.265265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.266405 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.266407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.267289 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.267289 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.267349 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.267354 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:40:54.267406 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:40:54.267452 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":34,"threads_started":1}
I20260926 22:40:54.267822 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.263160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.11.107.193:38915)
0926 22:40:54.263343 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:54.263348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:54.263761 (+ 413us) server_negotiation.cc:407] Connection header received
0926 22:40:54.263804 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:54.263807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:54.263822 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:54.263897 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:54.264020 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.265165 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.266485 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.267143 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.267205 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.267257 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:54.267675 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 22:40:54.267717 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
I20260926 22:40:54.268057 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.262952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.11.107.195:33887)
0926 22:40:54.264040 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:54.264046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:54.264056 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:40:54.264093 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:54.264094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:54.264108 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:54.264186 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:54.264350 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.265361 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.266567 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.267332 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.267406 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.267919 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:54.267946 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 22:40:54.267986 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052,"thread_start_us":105,"threads_started":1}
I20260926 22:40:54.267858 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.262847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:33887 (local address 127.0.0.1:49830)
0926 22:40:54.263134 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:54.263186 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:54.263246 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:54.264238 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:54.264240 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:54.264241 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:54.264276 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.264277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.265453 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.265455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.266497 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.266500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.267628 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.267629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.267702 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.267707 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:40:54.267756 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:40:54.267790 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":57,"threads_started":1}
I20260926 22:40:54.268266 22180 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae2e98051914672841c5c6d096f9c10 (DEFAULT_TABLE table=test-table [id=588a9eb4aa264bb0b53e76df4bf4a107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:54.268628 22180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae2e98051914672841c5c6d096f9c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:54.268584 22046 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae2e98051914672841c5c6d096f9c10 (DEFAULT_TABLE table=test-table [id=588a9eb4aa264bb0b53e76df4bf4a107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:54.268761 22314 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae2e98051914672841c5c6d096f9c10 (DEFAULT_TABLE table=test-table [id=588a9eb4aa264bb0b53e76df4bf4a107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:40:54.268886 22046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae2e98051914672841c5c6d096f9c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:54.269045 22314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae2e98051914672841c5c6d096f9c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:40:54.271937 22394 tablet_bootstrap.cc:492] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e: Bootstrap starting.
I20260926 22:40:54.272459 22395 tablet_bootstrap.cc:492] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf: Bootstrap starting.
I20260926 22:40:54.272595 22394 tablet_bootstrap.cc:654] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:54.272954 22395 tablet_bootstrap.cc:654] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:54.273138 22394 log.cc:824] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:54.273350 22395 log.cc:824] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:54.273489 22396 tablet_bootstrap.cc:492] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd: Bootstrap starting.
I20260926 22:40:54.273921 22394 tablet_bootstrap.cc:492] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e: No bootstrap required, opened a new log
I20260926 22:40:54.274143 22395 tablet_bootstrap.cc:492] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf: No bootstrap required, opened a new log
I20260926 22:40:54.274106 22394 ts_tablet_manager.cc:1397] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:40:54.274226 22395 ts_tablet_manager.cc:1397] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:40:54.275525 22395 raft_consensus.cc:348] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ebc12ddc984a5db173fa5163c7c7cd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33887 } } peers { permanent_uuid: "ab418466dea54211924b9afe428df3bf" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39619 } } peers { permanent_uuid: "6565dd061c244bd8a6bfefffd271233e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38915 } }
I20260926 22:40:54.275692 22395 raft_consensus.cc:374] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:54.275738 22395 raft_consensus.cc:729] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab418466dea54211924b9afe428df3bf, State: Initialized, Role: FOLLOWER
I20260926 22:40:54.275856 22395 consensus_queue.cc:260] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ebc12ddc984a5db173fa5163c7c7cd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33887 } } peers { permanent_uuid: "ab418466dea54211924b9afe428df3bf" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39619 } } peers { permanent_uuid: "6565dd061c244bd8a6bfefffd271233e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38915 } }
I20260926 22:40:54.275812 22394 raft_consensus.cc:348] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ebc12ddc984a5db173fa5163c7c7cd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33887 } } peers { permanent_uuid: "ab418466dea54211924b9afe428df3bf" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39619 } } peers { permanent_uuid: "6565dd061c244bd8a6bfefffd271233e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38915 } }
I20260926 22:40:54.276063 22394 raft_consensus.cc:374] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:54.276119 22394 raft_consensus.cc:729] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6565dd061c244bd8a6bfefffd271233e, State: Initialized, Role: FOLLOWER
I20260926 22:40:54.276226 22395 ts_tablet_manager.cc:1428] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 22:40:54.276264 22246 heartbeater.cc:503] Master 127.11.107.254:39955 was elected leader, sending a full tablet report...
I20260926 22:40:54.276260 22394 consensus_queue.cc:260] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ebc12ddc984a5db173fa5163c7c7cd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33887 } } peers { permanent_uuid: "ab418466dea54211924b9afe428df3bf" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39619 } } peers { permanent_uuid: "6565dd061c244bd8a6bfefffd271233e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38915 } }
I20260926 22:40:54.276554 22394 ts_tablet_manager.cc:1428] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:40:54.276573 22112 heartbeater.cc:503] Master 127.11.107.254:39955 was elected leader, sending a full tablet report...
I20260926 22:40:54.277227 22396 tablet_bootstrap.cc:654] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd: Neither blocks nor log segments found. Creating new log.
I20260926 22:40:54.277791 22396 log.cc:824] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd: Log is configured to *not* fsync() on all Append() calls
I20260926 22:40:54.278571 22396 tablet_bootstrap.cc:492] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd: No bootstrap required, opened a new log
I20260926 22:40:54.278643 22396 ts_tablet_manager.cc:1397] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260926 22:40:54.279937 22396 raft_consensus.cc:348] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ebc12ddc984a5db173fa5163c7c7cd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33887 } } peers { permanent_uuid: "ab418466dea54211924b9afe428df3bf" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39619 } } peers { permanent_uuid: "6565dd061c244bd8a6bfefffd271233e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38915 } }
I20260926 22:40:54.280055 22396 raft_consensus.cc:374] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:40:54.280076 22396 raft_consensus.cc:729] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ebc12ddc984a5db173fa5163c7c7cd, State: Initialized, Role: FOLLOWER
I20260926 22:40:54.280195 22396 consensus_queue.cc:260] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ebc12ddc984a5db173fa5163c7c7cd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33887 } } peers { permanent_uuid: "ab418466dea54211924b9afe428df3bf" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39619 } } peers { permanent_uuid: "6565dd061c244bd8a6bfefffd271233e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38915 } }
I20260926 22:40:54.280465 22396 ts_tablet_manager.cc:1428] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 22:40:54.280517 22380 heartbeater.cc:503] Master 127.11.107.254:39955 was elected leader, sending a full tablet report...
W20260926 22:40:54.330063 22247 tablet.cc:2369] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:40:54.386593 22399 raft_consensus.cc:482] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:40:54.386765 22399 raft_consensus.cc:504] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ebc12ddc984a5db173fa5163c7c7cd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33887 } } peers { permanent_uuid: "ab418466dea54211924b9afe428df3bf" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39619 } } peers { permanent_uuid: "6565dd061c244bd8a6bfefffd271233e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38915 } }
I20260926 22:40:54.387163 22399 leader_election.cc:288] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2ebc12ddc984a5db173fa5163c7c7cd (127.11.107.195:33887), 6565dd061c244bd8a6bfefffd271233e (127.11.107.193:38915)
I20260926 22:40:54.390057 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.387417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:59075 (local address 127.11.107.195:33887)
0926 22:40:54.387484 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:54.387486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:54.387496 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:40:54.387526 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:54.387527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:54.387537 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:54.387623 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:54.387858 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.388390 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.389323 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.389692 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.389745 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.389752 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:54.389927 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 22:40:54.389986 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 22:40:54.390062 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.387338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:33887 (local address 127.11.107.194:59075)
0926 22:40:54.387395 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:54.387408 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:54.387451 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:54.387649 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:54.387650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:54.387651 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:54.387756 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.387759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.388471 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.388473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.389229 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.389231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.389773 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.389774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.389875 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.389878 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:40:54.389918 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 22:40:54.389965 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 22:40:54.390172 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.387577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:40225 (local address 127.11.107.193:38915)
0926 22:40:54.387623 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:54.387625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:54.387747 (+ 122us) server_negotiation.cc:407] Connection header received
0926 22:40:54.387777 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:54.387778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:54.387787 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:54.387844 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:40:54.387994 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.388472 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.389362 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.389703 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.389793 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.389800 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:54.390050 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 22:40:54.390096 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:40:54.390363 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.387479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:38915 (local address 127.11.107.194:40225)
0926 22:40:54.387671 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:54.387681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:40:54.387738 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:40:54.387857 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:40:54.387858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:40:54.387859 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:40:54.387927 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.387928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.388557 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.388557 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.389252 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:40:54.389253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.389812 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.389813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:40:54.389942 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.389963 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 22:40:54.390019 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 22:40:54.390055 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260926 22:40:54.390416 22334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae2e98051914672841c5c6d096f9c10" candidate_uuid: "ab418466dea54211924b9afe428df3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ebc12ddc984a5db173fa5163c7c7cd" is_pre_election: true
I20260926 22:40:54.390599 22334 raft_consensus.cc:2454] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab418466dea54211924b9afe428df3bf in term 0.
I20260926 22:40:54.390810 22134 leader_election.cc:302] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab418466dea54211924b9afe428df3bf, d2ebc12ddc984a5db173fa5163c7c7cd; no voters:
I20260926 22:40:54.390846 22066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae2e98051914672841c5c6d096f9c10" candidate_uuid: "ab418466dea54211924b9afe428df3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6565dd061c244bd8a6bfefffd271233e" is_pre_election: true
I20260926 22:40:54.390966 22399 raft_consensus.cc:2781] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:40:54.390982 22066 raft_consensus.cc:2454] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab418466dea54211924b9afe428df3bf in term 0.
I20260926 22:40:54.391013 22399 raft_consensus.cc:482] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:40:54.391029 22399 raft_consensus.cc:3037] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:54.391744 22399 raft_consensus.cc:504] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ebc12ddc984a5db173fa5163c7c7cd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33887 } } peers { permanent_uuid: "ab418466dea54211924b9afe428df3bf" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39619 } } peers { permanent_uuid: "6565dd061c244bd8a6bfefffd271233e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38915 } }
I20260926 22:40:54.391875 22399 leader_election.cc:288] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [CANDIDATE]: Term 1 election: Requested vote from peers d2ebc12ddc984a5db173fa5163c7c7cd (127.11.107.195:33887), 6565dd061c244bd8a6bfefffd271233e (127.11.107.193:38915)
I20260926 22:40:54.392035 22334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae2e98051914672841c5c6d096f9c10" candidate_uuid: "ab418466dea54211924b9afe428df3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ebc12ddc984a5db173fa5163c7c7cd"
I20260926 22:40:54.392098 22334 raft_consensus.cc:3037] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:54.392223 22066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae2e98051914672841c5c6d096f9c10" candidate_uuid: "ab418466dea54211924b9afe428df3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6565dd061c244bd8a6bfefffd271233e"
I20260926 22:40:54.392320 22066 raft_consensus.cc:3037] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:40:54.393018 22334 raft_consensus.cc:2454] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab418466dea54211924b9afe428df3bf in term 1.
I20260926 22:40:54.393174 22134 leader_election.cc:302] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab418466dea54211924b9afe428df3bf, d2ebc12ddc984a5db173fa5163c7c7cd; no voters:
I20260926 22:40:54.393285 22399 raft_consensus.cc:2781] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:40:54.393301 22066 raft_consensus.cc:2454] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab418466dea54211924b9afe428df3bf in term 1.
I20260926 22:40:54.393453 22399 raft_consensus.cc:686] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [term 1 LEADER]: Becoming Leader. State: Replica: ab418466dea54211924b9afe428df3bf, State: Running, Role: LEADER
I20260926 22:40:54.393559 22399 consensus_queue.cc:237] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ebc12ddc984a5db173fa5163c7c7cd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33887 } } peers { permanent_uuid: "ab418466dea54211924b9afe428df3bf" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39619 } } peers { permanent_uuid: "6565dd061c244bd8a6bfefffd271233e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38915 } }
I20260926 22:40:54.394454 21920 catalog_manager.cc:5495] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf reported cstate change: term changed from 0 to 1, leader changed from <none> to ab418466dea54211924b9afe428df3bf (127.11.107.194). New cstate: current_term: 1 leader_uuid: "ab418466dea54211924b9afe428df3bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ebc12ddc984a5db173fa5163c7c7cd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab418466dea54211924b9afe428df3bf" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6565dd061c244bd8a6bfefffd271233e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38915 } health_report { overall_health: UNKNOWN } } }
I20260926 22:40:54.410313 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.407853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.11.107.194:39619)
0926 22:40:54.407895 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:40:54.407897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:40:54.407904 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:40:54.407968 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:40:54.407968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:40:54.407984 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:40:54.408063 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:40:54.408221 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.408684 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.409471 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:40:54.409661 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:40:54.409769 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:40:54.409962 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:40:54.410060 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:40:54.410154 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 22:40:54.410187 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 22:40:54.410206 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:40:54.412106 22066 raft_consensus.cc:1261] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e [term 1 FOLLOWER]: Refusing update from remote peer ab418466dea54211924b9afe428df3bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:54.412110 22334 raft_consensus.cc:1261] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd [term 1 FOLLOWER]: Refusing update from remote peer ab418466dea54211924b9afe428df3bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:40:54.412465 22404 consensus_queue.cc:1035] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6565dd061c244bd8a6bfefffd271233e" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 38915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:54.412590 22399 consensus_queue.cc:1035] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2ebc12ddc984a5db173fa5163c7c7cd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:40:54.414974 22407 mvcc.cc:205] Tried to move back new op lower bound from 7333734213269184512 to 7333734213196701696. Current Snapshot: MvccSnapshot[applied={T|T < 7333734213269184512}]
I20260926 22:40:54.415297 22409 mvcc.cc:205] Tried to move back new op lower bound from 7333734213269184512 to 7333734213196701696. Current Snapshot: MvccSnapshot[applied={T|T < 7333734213269184512}]
I20260926 22:40:54.415469 22408 mvcc.cc:205] Tried to move back new op lower bound from 7333734213269184512 to 7333734213196701696. Current Snapshot: MvccSnapshot[applied={T|T < 7333734213269184512}]
I20260926 22:40:54.417778 21920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:54.417887 21920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:40:54.418634 21920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82013e4c3ad94f1686d00508b22e0a63: Sending DeleteTablet for 3 replicas of tablet 7ae2e98051914672841c5c6d096f9c10
I20260926 22:40:54.418984 22180 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae2e98051914672841c5c6d096f9c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:54 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:55586
I20260926 22:40:54.419014 22314 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae2e98051914672841c5c6d096f9c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:54 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:49830
I20260926 22:40:54.419195 22046 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae2e98051914672841c5c6d096f9c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:40:54 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:51178
I20260926 22:40:54.419279 22417 tablet_replica.cc:330] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd: stopping tablet replica
I20260926 22:40:54.419371 22417 raft_consensus.cc:2229] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:54.419453 22418 tablet_replica.cc:330] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e: stopping tablet replica
I20260926 22:40:54.419484 22417 raft_consensus.cc:2258] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:54.419531 22418 raft_consensus.cc:2229] T 7ae2e98051914672841c5c6d096f9c10 P 6565dd061c244bd8a6bfefffd271233e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:40:54.419636 22416 tablet_replica.cc:330] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf: stopping tablet replica
I20260926 22:40:54.419677 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 21983
I20260926 22:40:54.419729 22416 raft_consensus.cc:2229] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:40:54.419833 22416 raft_consensus.cc:2258] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:40:54.420590 22417 ts_tablet_manager.cc:1905] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:54.420768 22416 ts_tablet_manager.cc:1905] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:40:54.422281 22417 ts_tablet_manager.cc:1918] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:54.422346 22417 log.cc:1196] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/7ae2e98051914672841c5c6d096f9c10
I20260926 22:40:54.422369 22416 ts_tablet_manager.cc:1918] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:40:54.422425 22416 log.cc:1196] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/7ae2e98051914672841c5c6d096f9c10
I20260926 22:40:54.422626 22417 ts_tablet_manager.cc:1939] T 7ae2e98051914672841c5c6d096f9c10 P d2ebc12ddc984a5db173fa5163c7c7cd: Deleting consensus metadata
I20260926 22:40:54.422713 22416 ts_tablet_manager.cc:1939] T 7ae2e98051914672841c5c6d096f9c10 P ab418466dea54211924b9afe428df3bf: Deleting consensus metadata
I20260926 22:40:54.423033 21907 catalog_manager.cc:4841] TS ab418466dea54211924b9afe428df3bf (127.11.107.194:39619): tablet 7ae2e98051914672841c5c6d096f9c10 (table test-table [id=588a9eb4aa264bb0b53e76df4bf4a107]) successfully deleted
I20260926 22:40:54.423467 21906 catalog_manager.cc:4841] TS d2ebc12ddc984a5db173fa5163c7c7cd (127.11.107.195:33887): tablet 7ae2e98051914672841c5c6d096f9c10 (table test-table [id=588a9eb4aa264bb0b53e76df4bf4a107]) successfully deleted
W20260926 22:40:54.427548 21906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:38915 (error 108)
I20260926 22:40:54.427798 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 22117
I20260926 22:40:54.428460 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:40:54.427926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:38915 (local address 127.0.0.1:51194)
0926 22:40:54.428384 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 22:40:54.428422 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440}
W20260926 22:40:54.428620 21906 catalog_manager.cc:4584] TS 6565dd061c244bd8a6bfefffd271233e (127.11.107.193:38915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ae2e98051914672841c5c6d096f9c10: Network error: Client connection negotiation failed: client connection to 127.11.107.193:38915: connect: Connection refused (error 111)
I20260926 22:40:54.433249 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 22251
I20260926 22:40:54.437425 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 21889
2026-09-26T22:40:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1953 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:40:54 dist-test-slave-j8sq krb5kdc[22422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:40:54 dist-test-slave-j8sq krb5kdc[22422](info): set up 2 sockets
Sep 26 22:40:54 dist-test-slave-j8sq krb5kdc[22422](info): commencing operation
krb5kdc: starting...
W20260926 22:40:55.467938 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:40:55 dist-test-slave-j8sq krb5kdc[22422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:40:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:40:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.107.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.107.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/rangeradmin_127.11.107.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/rangeradmin_127.11.107.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.107.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.107.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/rangerlookup_127.11.107.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/rangerlookup_127.11.107.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.107.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.107.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/HTTP_127.11.107.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/HTTP_127.11.107.212@KRBTEST.COM.keytab.
I20260926 22:40:55.618561 11695 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4c2jnj/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/postgres -l logfile start
2026-09-26 22:40:57.962 UTC [22452] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 22:40:57.962 UTC [22452] LOG: listening on IPv4 address "127.11.107.212", port 35299
2026-09-26 22:40:57.965 UTC [22452] LOG: listening on Unix socket "/tmp/.s.PGSQL.35299"
2026-09-26 22:40:57.986 UTC [22455] LOG: database system was shut down at 2026-09-26 22:40:56 UTC
2026-09-26 22:40:57.989 UTC [22452] LOG: database system is ready to accept connections
I20260926 22:40:58.943747 11695 mini_postgres.cc:96] Postgres bound to 35299
2026-09-26 22:40:58.949 UTC [22463] FATAL: database "slave" does not exist
127.11.107.212:35299 - accepting connections
I20260926 22:40:58.950234 11695 mini_ranger.cc:161] Starting Ranger...
I20260926 22:40:58.964550 11695 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 22:40:59.415162 11695 mini_ranger.cc:87] Created ranger Postgres database
I20260926 22:40:59.415261 11695 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-admin
2026-09-26 22:40:59,724 [I] DB FLAVOR :POSTGRES
2026-09-26 22:40:59,725 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 22:40:59,725 [I] Checking connection..
2026-09-26 22:40:59,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 22:40:59,995 [I] Checking connection passed.
2026-09-26 22:40:59,996 [I] --------- Verifying version history table ---------
2026-09-26 22:40:59,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 22:41:00,242 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 22:41:00,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 22:41:00,467 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 22:41:00,468 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 22:41:00,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 22:41:00.697 UTC [22527] WARNING: there is no transaction in progress
2026-09-26 22:41:00,703 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 22:41:00,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 22:41:00,960 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 22:41:00,961 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 22:41:00,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 22:41:01,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 22:41:01,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j8sq.c.gcp-upstream.internal','N') ;"
2026-09-26 22:41:01,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 22:41:01,920 [I] Table x_portal_user does not exist in database ranger
2026-09-26 22:41:01,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 22:41:02,174 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 22:41:02,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 22:41:02,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 22:41:02,653 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 22:41:02,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 22:41:03.576 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.590 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.607 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.621 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.636 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.652 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.704 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.710 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.723 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.732 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.741 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.754 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.763 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.769 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.775 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:03.781 UTC [22635] WARNING: there is no transaction in progress
2026-09-26 22:41:04,374 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 22:41:04,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 22:41:04,613 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 22:41:04,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 22:41:04,842 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 22:41:04,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 22:41:05,079 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 22:41:05,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 22:41:05,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 22:41:05,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j8sq.c.gcp-upstream.internal';"
2026-09-26 22:41:05,828 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 22:41:05,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 22:41:06,050 [I] DB_PATCHES have already been applied
I20260926 22:41:06.056092 11695 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-admin:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4c2jnj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 22:41:06.056156 11695 mini_ranger.cc:193] Using host: 127.11.107.212
I20260926 22:41:06.059098 11695 mini_ranger.cc:227] Ranger admin URL: http://127.11.107.212:35179
Sep 26, 2026 10:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 10:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 10:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 10:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 10:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 10:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 10:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 22:41:07.163566 11695 mini_ranger.cc:160] Time spent starting Ranger: real 8.213s user 0.003s sys 0.005s
Sep 26, 2026 10:41:26 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 10:41:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 10:41:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 10:41:26 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 10:41:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 10:41:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 10:41:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 10:41:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 10:41:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 10:41:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 10:41:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 10:41:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 10:41:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 10:41:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 10:41:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 10:41:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 10:41:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 10:41:27 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 22:41:28.505666 11695 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.107.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.107.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/rangerkms_127.11.107.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/rangerkms_127.11.107.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.107.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.107.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.107.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/HTTP_127.11.107.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.107.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/HTTP_127.11.107.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 22:41:28 dist-test-slave-j8sq krb5kdc[22422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462488, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 22:41:28.591383 11695 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 22:41:28.599532 11695 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 22:41:29.365270 11695 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 22:41:29.365355 11695 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-kms
I20260926 22:41:29.365377 11695 mini_ranger_kms.cc:227] Using postgres at 127.11.107.212:35299
2026-09-26 22:41:29,541 [I] DB FLAVOR :POSTGRES
2026-09-26 22:41:29,542 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 22:41:29,542 [I] Checking connection
2026-09-26 22:41:29,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 22:41:29,767 [I] connection success
2026-09-26 22:41:29,767 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 22:41:29,767 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 22:41:29,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 22:41:29,990 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 22:41:29,990 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 22:41:29,990 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 22:41:29,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35299/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 22:41:30,236 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 22:41:30.682425 11695 mini_ranger_kms.cc:310] Created kms service
I20260926 22:41:30.871130 11695 mini_ranger_kms.cc:326] Created kudu user
I20260926 22:41:30.916898 11695 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 22:41:30 dist-test-slave-j8sq krb5kdc[22422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462488, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:41:31.306267 11695 mini_ranger_kms.cc:384] Added ranger policy
I20260926 22:41:31.306376 11695 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-kms:/tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 22:41:31.306396 11695 mini_ranger_kms.cc:240] Using host: 127.11.107.212
I20260926 22:41:31.309674 11695 mini_ranger_kms.cc:276] Ranger KMS PID: 22815
I20260926 22:41:31.309801 11695 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.107.212:48793
Sep 26, 2026 10:41:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 10:41:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 10:41:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 10:41:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 10:41:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 10:41:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 22:41:33.586426 11695 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.995s user 0.006s sys 0.153s
I20260926 22:41:33.586546 11695 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 22:41:33.586647 11695 mini_ranger_kms.cc:401] 127.11.107.212:48793/kms/v1/keys
Sep 26 22:41:33 dist-test-slave-j8sq krb5kdc[22422](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790462493, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.107.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 22:41:34 dist-test-slave-j8sq krb5kdc[22422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462493, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.107.212@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
Sep 26 22:41:34 dist-test-slave-j8sq krb5kdc[22422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462493, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.107.212@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
Sep 26 22:41:36 dist-test-slave-j8sq krb5kdc[22422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:41:36.538643 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:43623
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:43623
--ranger_config_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.107.212:48793/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:41:36.642614 22866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:41:36.642885 22866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:41:36.642952 22866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:41:36.644847 22866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:41:36.644899 22866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:41:36.644922 22866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:41:36.644943 22866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:41:36.644991 22866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:41:36.646962 22866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.107.212:48793/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:43623
--ranger_config_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:43623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:41:36.647294 22866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:41:36.647529 22866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:41:36.653541 22866 server_base.cc:1034] running on GCE node
W20260926 22:41:36.653519 22871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:41:36.653560 22872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:41:36.653527 22874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:41:36.654043 22866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:41:36.654389 22866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:41:36.655560 22866 hybrid_clock.cc:648] HybridClock initialized: now 1790462496655546 us; error 27 us; skew 500 ppm
Sep 26 22:41:36 dist-test-slave-j8sq krb5kdc[22422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462496, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:41:36.658715 22866 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:41:36.659148 22866 webserver.cc:460] Webserver started at http://127.11.107.254:37673/ using document root <none> and password file <none>
I20260926 22:41:36.659384 22866 fs_manager.cc:359] Metadata directory not provided
I20260926 22:41:36.659442 22866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:41:36.659580 22866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:41:36 dist-test-slave-j8sq krb5kdc[22422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462496, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:41:36.714145 22866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "8e6f6140b09247f58bcfd523ba27d559"
format_stamp: "Formatted at 2026-09-26 22:41:36 on dist-test-slave-j8sq"
server_key: "82ea3a906b8cbd53de79c745fa39e3f8"
server_key_iv: "33c107dca48be1ffa3dd1dc8baf4e46a"
server_key_version: "kuduclusterkey@0"
I20260926 22:41:36.714600 22866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "8e6f6140b09247f58bcfd523ba27d559"
format_stamp: "Formatted at 2026-09-26 22:41:36 on dist-test-slave-j8sq"
server_key: "82ea3a906b8cbd53de79c745fa39e3f8"
server_key_iv: "33c107dca48be1ffa3dd1dc8baf4e46a"
server_key_version: "kuduclusterkey@0"
I20260926 22:41:36.716320 22866 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 22:41:36.731020 22881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:41:36.731276 22866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 22:41:36.731389 22866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "8e6f6140b09247f58bcfd523ba27d559"
format_stamp: "Formatted at 2026-09-26 22:41:36 on dist-test-slave-j8sq"
server_key: "82ea3a906b8cbd53de79c745fa39e3f8"
server_key_iv: "33c107dca48be1ffa3dd1dc8baf4e46a"
server_key_version: "kuduclusterkey@0"
I20260926 22:41:36.731483 22866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:41:36.745827 22866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:41:36.747776 22866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:41:36.747918 22866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:41:36.752723 22866 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:43623
I20260926 22:41:36.752790 22933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:43623 every 8 connection(s)
I20260926 22:41:36.753140 22866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:41:36.753818 22934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:41:36.754832 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 22866
I20260926 22:41:36.754940 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:41:36.755131 11695 external_mini_cluster.cc:1428] Setting key a8c010ba41a69779f453ed6fd013c9d2
I20260926 22:41:36.757087 22934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559: Bootstrap starting.
I20260926 22:41:36.757786 22934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559: Neither blocks nor log segments found. Creating new log.
I20260926 22:41:36.758104 22934 log.cc:824] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559: Log is configured to *not* fsync() on all Append() calls
I20260926 22:41:36.758867 22934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559: No bootstrap required, opened a new log
I20260926 22:41:36.761046 22934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6f6140b09247f58bcfd523ba27d559" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43623 } }
I20260926 22:41:36.761242 22934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:41:36.761291 22934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e6f6140b09247f58bcfd523ba27d559, State: Initialized, Role: FOLLOWER
I20260926 22:41:36.761431 22934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6f6140b09247f58bcfd523ba27d559" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43623 } }
I20260926 22:41:36.761526 22934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:41:36.761564 22934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:41:36.761603 22934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:41:36.762253 22934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6f6140b09247f58bcfd523ba27d559" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43623 } }
I20260926 22:41:36.762349 22934 leader_election.cc:302] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e6f6140b09247f58bcfd523ba27d559; no voters:
I20260926 22:41:36.762485 22934 leader_election.cc:288] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:41:36.762694 22934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 22:41:36 dist-test-slave-j8sq krb5kdc[22422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:41:36.763345 22940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:41:36.763507 22940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [term 1 LEADER]: Becoming Leader. State: Replica: 8e6f6140b09247f58bcfd523ba27d559, State: Running, Role: LEADER
I20260926 22:41:36.763577 22940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6f6140b09247f58bcfd523ba27d559" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43623 } }
I20260926 22:41:36.763873 22940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e6f6140b09247f58bcfd523ba27d559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6f6140b09247f58bcfd523ba27d559" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43623 } } }
I20260926 22:41:36.763936 22940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [sys.catalog]: This master's current role is: LEADER
I20260926 22:41:36.764045 22940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e6f6140b09247f58bcfd523ba27d559. Latest consensus state: current_term: 1 leader_uuid: "8e6f6140b09247f58bcfd523ba27d559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6f6140b09247f58bcfd523ba27d559" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43623 } } }
I20260926 22:41:36.764091 22940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559 [sys.catalog]: This master's current role is: LEADER
I20260926 22:41:36.764219 22934 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 22:41:36.768133 22953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:41:36.768919 22953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:41:36.770318 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:36.755622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49748 (local address 127.11.107.254:43623)
0926 22:41:36.755811 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 22:41:36.755820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:41:36.755851 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:41:36.756366 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:41:36.756374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:41:36.756440 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:41:36.756591 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:41:36.757356 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:36.758725 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:36.759816 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:36.760499 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:36.760619 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:36.763329 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:41:36.763335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:41:36.763335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:41:36.763348 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:41:36.767907 (+ 4559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:41:36.768374 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:41:36.768377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:41:36.768378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:41:36.768550 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:41:36.768778 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:41:36.768779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:41:36.768779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:41:36.769214 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:41:36.769457 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 22:41:36.769547 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 22:41:36.769982 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260926 22:41:36.774283 22953 catalog_manager.cc:1347] Generated new cluster ID: c87e0d1516694533a46ad71264c484fe
I20260926 22:41:36.774377 22953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:41:36.788792 22953 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:41:36.789536 22953 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:41:36.795526 22953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559: Generated new TSK 0
I20260926 22:41:36.795782 22953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:41:37.486327 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.107.212:48793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:43623
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:41:37.571774 22979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:41:37.571995 22979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:41:37.572031 22979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:41:37.573156 22979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:41:37.573221 22979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:41:37.573264 22979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:41:37.574795 22979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.107.212:48793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:41:37.575033 22979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:41:37.575265 22979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:41:37.580082 22984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:41:37.580252 22987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:41:37.580108 22985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:41:37.580583 22979 server_base.cc:1034] running on GCE node
I20260926 22:41:37.580817 22979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:41:37.581126 22979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:41:37.582341 22979 hybrid_clock.cc:648] HybridClock initialized: now 1790462497582322 us; error 55 us; skew 500 ppm
Sep 26 22:41:37 dist-test-slave-j8sq krb5kdc[22422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462497, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:41:37.585472 22979 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:41:37.585991 22979 webserver.cc:460] Webserver started at http://127.11.107.193:40145/ using document root <none> and password file <none>
I20260926 22:41:37.586226 22979 fs_manager.cc:359] Metadata directory not provided
I20260926 22:41:37.586292 22979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:41:37.586426 22979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:41:37 dist-test-slave-j8sq krb5kdc[22422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462497, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:41:37.598992 22979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "0f020c35b1c3457589c04c04d6a72165"
format_stamp: "Formatted at 2026-09-26 22:41:37 on dist-test-slave-j8sq"
server_key: "1352245c90121b629737671be945ef22"
server_key_iv: "731b89c05a07dc7a9ec97db30dad22c6"
server_key_version: "kuduclusterkey@0"
I20260926 22:41:37.599673 22979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "0f020c35b1c3457589c04c04d6a72165"
format_stamp: "Formatted at 2026-09-26 22:41:37 on dist-test-slave-j8sq"
server_key: "1352245c90121b629737671be945ef22"
server_key_iv: "731b89c05a07dc7a9ec97db30dad22c6"
server_key_version: "kuduclusterkey@0"
I20260926 22:41:37.601778 22979 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 22:41:37.615717 22994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:41:37.615908 22979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:41:37.616017 22979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "0f020c35b1c3457589c04c04d6a72165"
format_stamp: "Formatted at 2026-09-26 22:41:37 on dist-test-slave-j8sq"
server_key: "1352245c90121b629737671be945ef22"
server_key_iv: "731b89c05a07dc7a9ec97db30dad22c6"
server_key_version: "kuduclusterkey@0"
I20260926 22:41:37.616104 22979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:41:37.640130 22979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:41:37.642583 22979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:41:37.642743 22979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:41:37.642973 22979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:41:37.643378 22979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:41:37.643422 22979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:41:37.643452 22979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:41:37.643473 22979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:41:37.650772 22979 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:45825
I20260926 22:41:37.651127 22979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:41:37.651479 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 22979
I20260926 22:41:37.651593 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:41:37.651773 11695 external_mini_cluster.cc:1428] Setting key 39780e76ba383148bd1d4d31c36fc508
I20260926 22:41:37.653741 23109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:45825 every 8 connection(s)
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Sep 26 22:41:37 dist-test-slave-j8sq krb5kdc[22422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462497, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:41:37.670526 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:37.651912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:43623 (local address 127.11.107.193:40837)
0926 22:41:37.652436 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 22:41:37.652549 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0926 22:41:37.653154 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:41:37.655613 (+ 2459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:41:37.655618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:41:37.655856 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:41:37.656120 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:37.656127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:37.657171 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:37.657172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:37.663813 (+ 6641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:37.663817 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:37.664397 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:37.664398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:37.664548 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:37.665135 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:41:37.665146 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:41:37.666791 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:41:37.668839 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:41:37.668846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:41:37.668849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:41:37.669127 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:41:37.669365 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:41:37.669366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:41:37.669366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:41:37.669492 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:41:37.669791 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:41:37.669797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:41:37.670012 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 22:41:37.670125 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 22:41:37.670288 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":71,"threads_started":1}
I20260926 22:41:37.670964 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:37.652130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:40837 (local address 127.11.107.254:43623)
0926 22:41:37.655379 (+ 3249us) server_negotiation.cc:206] Beginning negotiation
0926 22:41:37.655382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:41:37.655391 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:41:37.655417 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:41:37.655418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:41:37.655429 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:41:37.656231 (+ 802us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:41:37.656271 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:37.657061 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:37.663929 (+ 6868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:37.664269 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:37.664329 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:37.666899 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:41:37.666900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:41:37.666901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:41:37.666908 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:41:37.668743 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:41:37.669225 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:41:37.669227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:41:37.669227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:41:37.669295 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:41:37.669568 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:41:37.669569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:41:37.669569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:41:37.669701 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:41:37.669764 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 22:41:37.670786 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0926 22:41:37.670870 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3220,"thread_start_us":91,"threads_started":1}
I20260926 22:41:37.671319 23110 heartbeater.cc:347] Connected to a master server at 127.11.107.254:43623
I20260926 22:41:37.671406 23110 heartbeater.cc:464] Registering TS with master...
I20260926 22:41:37.671582 23110 heartbeater.cc:511] Master 127.11.107.254:43623 requested a full tablet report, sending...
I20260926 22:41:37.676539 22898 ts_manager.cc:194] Registered new tserver with Master: 0f020c35b1c3457589c04c04d6a72165 (127.11.107.193:45825)
I20260926 22:41:37.677891 22898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:40837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:41:37.690927 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.107.212:48793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:43623
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:41:37.805895 23116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:41:37.806151 23116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:41:37.806183 23116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:41:37.807824 23116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:41:37.807873 23116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:41:37.807907 23116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:41:37.809994 23116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.107.212:48793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:41:37.810220 23116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:41:37.810449 23116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:41:37.815856 23126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:41:37.816453 23124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:41:37.816458 23123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:41:37.816941 23116 server_base.cc:1034] running on GCE node
I20260926 22:41:37.817147 23116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:41:37.817442 23116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:41:37.818591 23116 hybrid_clock.cc:648] HybridClock initialized: now 1790462497818579 us; error 28 us; skew 500 ppm
Sep 26 22:41:37 dist-test-slave-j8sq krb5kdc[22422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462497, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:41:37.821656 23116 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:41:37.822139 23116 webserver.cc:460] Webserver started at http://127.11.107.194:37387/ using document root <none> and password file <none>
I20260926 22:41:37.822378 23116 fs_manager.cc:359] Metadata directory not provided
I20260926 22:41:37.822448 23116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:41:37.822585 23116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:41:37 dist-test-slave-j8sq krb5kdc[22422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462497, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
Sep 26 22:41:37 dist-test-slave-j8sq krb5kdc[22422](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790462497, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:41:37.838546 23116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "632318d2043c42b097640fc9786a882e"
format_stamp: "Formatted at 2026-09-26 22:41:37 on dist-test-slave-j8sq"
server_key: "c1e8b3d53765e37514bd9f389d0ba748"
server_key_iv: "f2996f042fde1af9205420ba68e3c7a0"
server_key_version: "kuduclusterkey@0"
I20260926 22:41:37.839010 23116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "632318d2043c42b097640fc9786a882e"
format_stamp: "Formatted at 2026-09-26 22:41:37 on dist-test-slave-j8sq"
server_key: "c1e8b3d53765e37514bd9f389d0ba748"
server_key_iv: "f2996f042fde1af9205420ba68e3c7a0"
server_key_version: "kuduclusterkey@0"
I20260926 22:41:37.842015 23116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 22:41:37.874039 23133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:41:37.874305 23116 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260926 22:41:37.874420 23116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "632318d2043c42b097640fc9786a882e"
format_stamp: "Formatted at 2026-09-26 22:41:37 on dist-test-slave-j8sq"
server_key: "c1e8b3d53765e37514bd9f389d0ba748"
server_key_iv: "f2996f042fde1af9205420ba68e3c7a0"
server_key_version: "kuduclusterkey@0"
I20260926 22:41:37.874538 23116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:41:37.889336 23116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:41:37.891448 23116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:41:37.891608 23116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:41:37.891907 23116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:41:37.892285 23116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:41:37.892347 23116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:41:37.892385 23116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:41:37.892412 23116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:41:37.899479 23116 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:34379
I20260926 22:41:37.899593 23246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:34379 every 8 connection(s)
I20260926 22:41:37.899864 23116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:41:37 dist-test-slave-j8sq krb5kdc[22422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462497, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:41:37.909682 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:37.900633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:37993 (local address 127.11.107.254:43623)
0926 22:41:37.900714 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 22:41:37.900717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:41:37.900964 (+ 247us) server_negotiation.cc:407] Connection header received
0926 22:41:37.901487 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:41:37.901488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:41:37.901501 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:41:37.901557 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:41:37.902166 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:37.902660 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:37.903639 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:37.903857 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:37.903931 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:37.906365 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:41:37.906366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:41:37.906366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:41:37.906374 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:41:37.908068 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:41:37.908541 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:41:37.908542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:41:37.908542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:41:37.908605 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:41:37.908910 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:41:37.908911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:41:37.908912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:41:37.909051 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:41:37.909124 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 22:41:37.909475 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 22:41:37.909581 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 22:41:37.910135 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 23116
I20260926 22:41:37.910245 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:41:37.910437 11695 external_mini_cluster.cc:1428] Setting key ebc299ff1d4fc95f3e97b512b7218d62
I20260926 22:41:37.911170 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:37.900614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:43623 (local address 127.11.107.194:37993)
0926 22:41:37.900857 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 22:41:37.900886 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 22:41:37.901384 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:41:37.901631 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:41:37.901635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:41:37.901818 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:41:37.902062 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:37.902067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:37.902752 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:37.902753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:37.903533 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:37.903535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:37.903936 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:37.903938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:37.904077 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:37.904584 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:41:37.904592 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:41:37.906255 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:41:37.908165 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:41:37.908171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:41:37.908173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:41:37.908455 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:41:37.908689 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:41:37.908690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:41:37.908690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:41:37.908830 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:41:37.909133 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:41:37.909141 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:41:37.909301 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 22:41:37.910783 (+ 1482us) client_negotiation.cc:240] Negotiation successful
0926 22:41:37.910942 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
I20260926 22:41:37.911824 23247 heartbeater.cc:347] Connected to a master server at 127.11.107.254:43623
I20260926 22:41:37.911914 23247 heartbeater.cc:464] Registering TS with master...
I20260926 22:41:37.912092 23247 heartbeater.cc:511] Master 127.11.107.254:43623 requested a full tablet report, sending...
I20260926 22:41:37.912539 22898 ts_manager.cc:194] Registered new tserver with Master: 632318d2043c42b097640fc9786a882e (127.11.107.194:34379)
I20260926 22:41:37.913020 22898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:37993
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:41:37.938582 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.107.212:48793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:43623
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:41:38.031535 23253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:41:38.031757 23253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:41:38.031796 23253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:41:38.033020 23253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:41:38.033068 23253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:41:38.033109 23253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:41:38.034680 23253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.107.212:48793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:41:38.034940 23253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:41:38.035245 23253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:41:38.040644 23258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:41:38.040656 23261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:41:38.040904 23259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:41:38.041208 23253 server_base.cc:1034] running on GCE node
I20260926 22:41:38.041421 23253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:41:38.041695 23253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:41:38.042842 23253 hybrid_clock.cc:648] HybridClock initialized: now 1790462498042830 us; error 30 us; skew 500 ppm
Sep 26 22:41:38 dist-test-slave-j8sq krb5kdc[22422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462498, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:41:38.045460 23253 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:41:38.045995 23253 webserver.cc:460] Webserver started at http://127.11.107.195:38029/ using document root <none> and password file <none>
I20260926 22:41:38.046238 23253 fs_manager.cc:359] Metadata directory not provided
I20260926 22:41:38.046307 23253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:41:38.046443 23253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:41:38 dist-test-slave-j8sq krb5kdc[22422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462498, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:41:38.057406 23253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "de68f210f8354027bc7108276f65c973"
format_stamp: "Formatted at 2026-09-26 22:41:38 on dist-test-slave-j8sq"
server_key: "4acf93bbc63f7b878cbd97aa6fd08936"
server_key_iv: "931e7d071f2975151aacd49bd6913339"
server_key_version: "kuduclusterkey@0"
I20260926 22:41:38.057909 23253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "de68f210f8354027bc7108276f65c973"
format_stamp: "Formatted at 2026-09-26 22:41:38 on dist-test-slave-j8sq"
server_key: "4acf93bbc63f7b878cbd97aa6fd08936"
server_key_iv: "931e7d071f2975151aacd49bd6913339"
server_key_version: "kuduclusterkey@0"
I20260926 22:41:38.074329 23253 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.000s sys 0.001s
I20260926 22:41:38.098172 23268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:41:38.104946 23253 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.001s
I20260926 22:41:38.105088 23253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "de68f210f8354027bc7108276f65c973"
format_stamp: "Formatted at 2026-09-26 22:41:38 on dist-test-slave-j8sq"
server_key: "4acf93bbc63f7b878cbd97aa6fd08936"
server_key_iv: "931e7d071f2975151aacd49bd6913339"
server_key_version: "kuduclusterkey@0"
I20260926 22:41:38.105218 23253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:41:38.120515 23253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:41:38.123029 23253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:41:38.123178 23253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:41:38.123436 23253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:41:38.123785 23253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:41:38.123831 23253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:41:38.123862 23253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:41:38.123885 23253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:41:38.130816 23253 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:41139
I20260926 22:41:38.131304 23253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:41:38.131727 23381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:41139 every 8 connection(s)
I20260926 22:41:38.135214 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 23253
I20260926 22:41:38.135331 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:41:38.135550 11695 external_mini_cluster.cc:1428] Setting key 60e5b991ec1551ada697bd8045faa31c
Sep 26 22:41:38 dist-test-slave-j8sq krb5kdc[22422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462498, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:41:38.144057 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.132299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:46505 (local address 127.11.107.254:43623)
0926 22:41:38.132356 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:41:38.132358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:41:38.132389 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:41:38.133208 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:41:38.133211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:41:38.133224 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:41:38.133261 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:41:38.134174 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.134955 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.137289 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.137591 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.137672 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.140374 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:41:38.140375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:41:38.140376 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:41:38.140385 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:41:38.142383 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:41:38.142858 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:41:38.142859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:41:38.142859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:41:38.142942 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:41:38.143235 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:41:38.143236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:41:38.143237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:41:38.143403 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:41:38.143500 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 22:41:38.143815 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 22:41:38.143931 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 22:41:38.144146 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.132051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:43623 (local address 127.11.107.195:46505)
0926 22:41:38.132233 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 22:41:38.132300 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 22:41:38.133053 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:41:38.133467 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:41:38.133471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:41:38.133735 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:41:38.134047 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:38.134055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.135584 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.135585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:38.137138 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:38.137141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.137696 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.137697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:38.137871 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.138468 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:41:38.138478 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:41:38.140267 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:41:38.142479 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:41:38.142485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:41:38.142487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:41:38.142769 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:41:38.143027 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:41:38.143027 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:41:38.143028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:41:38.143155 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:41:38.143491 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:41:38.143497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:41:38.143649 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 22:41:38.143766 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 22:41:38.143915 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260926 22:41:38.144994 23382 heartbeater.cc:347] Connected to a master server at 127.11.107.254:43623
I20260926 22:41:38.145182 23382 heartbeater.cc:464] Registering TS with master...
I20260926 22:41:38.145462 23382 heartbeater.cc:511] Master 127.11.107.254:43623 requested a full tablet report, sending...
I20260926 22:41:38.146050 22898 ts_manager.cc:194] Registered new tserver with Master: de68f210f8354027bc7108276f65c973 (127.11.107.195:41139)
I20260926 22:41:38.146633 22898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:46505
I20260926 22:41:38.147486 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:41:38.161218 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.152834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.11.107.254:43623)
0926 22:41:38.152881 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:41:38.152883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:41:38.153034 (+ 151us) server_negotiation.cc:407] Connection header received
0926 22:41:38.153145 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:41:38.153146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:41:38.153157 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:41:38.153191 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:41:38.154078 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.154766 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.155834 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.156113 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.156158 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.157899 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:41:38.157901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:41:38.157901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:41:38.157909 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:41:38.159555 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:41:38.160350 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:41:38.160351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:41:38.160351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:41:38.160407 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:41:38.160660 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:41:38.160661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:41:38.160661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:41:38.160769 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:41:38.160887 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 22:41:38.161028 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 22:41:38.161107 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:41:38.162954 22898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:41:38.164942 22898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:41:38.172923 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.167914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:34379 (local address 127.0.0.1:53486)
0926 22:41:38.168693 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0926 22:41:38.168734 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:41:38.168809 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:41:38.169143 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:41:38.169146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:41:38.169148 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:41:38.169217 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:38.169219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.170541 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.170543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:38.171600 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:38.171602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.172627 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.172628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:38.172709 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.172724 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 22:41:38.172802 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 22:41:38.172848 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":80,"threads_started":1}
I20260926 22:41:38.173506 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.168395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.11.107.194:34379)
0926 22:41:38.168613 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 22:41:38.168618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:41:38.168902 (+ 284us) server_negotiation.cc:407] Connection header received
0926 22:41:38.168953 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:41:38.168955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:41:38.168973 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:41:38.169049 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:41:38.169308 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.170416 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.171739 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.172524 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.172631 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.172699 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 22:41:38.173358 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 22:41:38.173408 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260926 22:41:38.174342 23181 tablet_service.cc:1467] Processing CreateTablet for tablet 2168c1342dc44c3daebe42c421587648 (DEFAULT_TABLE table=test-table [id=73339b6e38e1423593de8f51e965d8b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:41:38.174731 23181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2168c1342dc44c3daebe42c421587648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:41:38.178651 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.168159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:41139 (local address 127.0.0.1:50758)
0926 22:41:38.169187 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0926 22:41:38.169199 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:41:38.169246 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:41:38.172998 (+ 3752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:41:38.172999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:41:38.173001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:41:38.173072 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:38.173074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.175024 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.175025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:38.176402 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:38.176404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.178419 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.178420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:38.178488 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.178492 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:41:38.178545 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 22:41:38.178587 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":64,"threads_started":1}
I20260926 22:41:38.179062 23400 tablet_bootstrap.cc:492] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e: Bootstrap starting.
I20260926 22:41:38.180266 23400 tablet_bootstrap.cc:654] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e: Neither blocks nor log segments found. Creating new log.
I20260926 22:41:38.180341 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.168754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.11.107.193:45825)
0926 22:41:38.171724 (+ 2970us) server_negotiation.cc:206] Beginning negotiation
0926 22:41:38.171727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:41:38.171737 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:41:38.171774 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:41:38.171776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:41:38.171789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:41:38.171881 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:41:38.172045 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.172919 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.178778 (+ 5859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.179579 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.179639 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.179690 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:41:38.180214 (+ 524us) server_negotiation.cc:299] Negotiation successful
0926 22:41:38.180264 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2939,"thread_start_us":52,"threads_started":1}
I20260926 22:41:38.180606 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.168494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:45825 (local address 127.0.0.1:58358)
0926 22:41:38.169332 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 22:41:38.169342 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:41:38.169382 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:41:38.171892 (+ 2510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:41:38.171894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:41:38.171895 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:41:38.171935 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:38.171937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.173469 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.173470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:38.177304 (+ 3834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:38.177307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.180052 (+ 2745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.180053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:38.180118 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.180121 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:41:38.180474 (+ 353us) client_negotiation.cc:240] Negotiation successful
0926 22:41:38.180535 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":66,"threads_started":1}
I20260926 22:41:38.180946 23400 log.cc:824] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e: Log is configured to *not* fsync() on all Append() calls
I20260926 22:41:38.181305 23043 tablet_service.cc:1467] Processing CreateTablet for tablet 2168c1342dc44c3daebe42c421587648 (DEFAULT_TABLE table=test-table [id=73339b6e38e1423593de8f51e965d8b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:41:38.181638 23043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2168c1342dc44c3daebe42c421587648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:41:38.181766 23400 tablet_bootstrap.cc:492] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e: No bootstrap required, opened a new log
I20260926 22:41:38.181815 23400 ts_tablet_manager.cc:1397] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 22:41:38.183739 23400 raft_consensus.cc:348] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f020c35b1c3457589c04c04d6a72165" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45825 } } peers { permanent_uuid: "632318d2043c42b097640fc9786a882e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34379 } } peers { permanent_uuid: "de68f210f8354027bc7108276f65c973" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 41139 } }
I20260926 22:41:38.183892 23400 raft_consensus.cc:374] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:41:38.183926 23400 raft_consensus.cc:729] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 632318d2043c42b097640fc9786a882e, State: Initialized, Role: FOLLOWER
I20260926 22:41:38.184041 23400 consensus_queue.cc:260] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f020c35b1c3457589c04c04d6a72165" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45825 } } peers { permanent_uuid: "632318d2043c42b097640fc9786a882e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34379 } } peers { permanent_uuid: "de68f210f8354027bc7108276f65c973" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 41139 } }
I20260926 22:41:38.184319 23400 ts_tablet_manager.cc:1428] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:41:38.184515 23247 heartbeater.cc:503] Master 127.11.107.254:43623 was elected leader, sending a full tablet report...
I20260926 22:41:38.185351 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.170865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.11.107.195:41139)
0926 22:41:38.172015 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0926 22:41:38.172019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:41:38.172029 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:41:38.172067 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:41:38.172069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:41:38.172087 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:41:38.172164 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:41:38.173896 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.174902 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.177424 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.178226 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.178321 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.178383 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 22:41:38.185208 (+ 6825us) server_negotiation.cc:299] Negotiation successful
0926 22:41:38.185254 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":85,"threads_started":1}
I20260926 22:41:38.186069 23403 tablet_bootstrap.cc:492] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165: Bootstrap starting.
I20260926 22:41:38.186043 23298 tablet_service.cc:1467] Processing CreateTablet for tablet 2168c1342dc44c3daebe42c421587648 (DEFAULT_TABLE table=test-table [id=73339b6e38e1423593de8f51e965d8b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:41:38.186362 23298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2168c1342dc44c3daebe42c421587648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:41:38.186687 23403 tablet_bootstrap.cc:654] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165: Neither blocks nor log segments found. Creating new log.
I20260926 22:41:38.186964 23403 log.cc:824] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165: Log is configured to *not* fsync() on all Append() calls
I20260926 22:41:38.189194 23405 tablet_bootstrap.cc:492] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973: Bootstrap starting.
I20260926 22:41:38.190393 23403 tablet_bootstrap.cc:492] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165: No bootstrap required, opened a new log
I20260926 22:41:38.190501 23403 ts_tablet_manager.cc:1397] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 22:41:38.192291 23405 tablet_bootstrap.cc:654] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973: Neither blocks nor log segments found. Creating new log.
I20260926 22:41:38.192608 23405 log.cc:824] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973: Log is configured to *not* fsync() on all Append() calls
I20260926 22:41:38.193212 23403 raft_consensus.cc:348] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f020c35b1c3457589c04c04d6a72165" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45825 } } peers { permanent_uuid: "632318d2043c42b097640fc9786a882e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34379 } } peers { permanent_uuid: "de68f210f8354027bc7108276f65c973" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 41139 } }
I20260926 22:41:38.193361 23403 raft_consensus.cc:374] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:41:38.193404 23403 raft_consensus.cc:729] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f020c35b1c3457589c04c04d6a72165, State: Initialized, Role: FOLLOWER
I20260926 22:41:38.193529 23403 consensus_queue.cc:260] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f020c35b1c3457589c04c04d6a72165" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45825 } } peers { permanent_uuid: "632318d2043c42b097640fc9786a882e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34379 } } peers { permanent_uuid: "de68f210f8354027bc7108276f65c973" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 41139 } }
I20260926 22:41:38.193807 23403 ts_tablet_manager.cc:1428] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:41:38.193869 23110 heartbeater.cc:503] Master 127.11.107.254:43623 was elected leader, sending a full tablet report...
I20260926 22:41:38.194591 23405 tablet_bootstrap.cc:492] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973: No bootstrap required, opened a new log
I20260926 22:41:38.194741 23405 ts_tablet_manager.cc:1397] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 22:41:38.196512 23405 raft_consensus.cc:348] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f020c35b1c3457589c04c04d6a72165" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45825 } } peers { permanent_uuid: "632318d2043c42b097640fc9786a882e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34379 } } peers { permanent_uuid: "de68f210f8354027bc7108276f65c973" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 41139 } }
I20260926 22:41:38.196659 23405 raft_consensus.cc:374] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:41:38.196702 23405 raft_consensus.cc:729] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de68f210f8354027bc7108276f65c973, State: Initialized, Role: FOLLOWER
I20260926 22:41:38.196837 23405 consensus_queue.cc:260] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f020c35b1c3457589c04c04d6a72165" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45825 } } peers { permanent_uuid: "632318d2043c42b097640fc9786a882e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34379 } } peers { permanent_uuid: "de68f210f8354027bc7108276f65c973" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 41139 } }
I20260926 22:41:38.197113 23405 ts_tablet_manager.cc:1428] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:41:38.197141 23382 heartbeater.cc:503] Master 127.11.107.254:43623 was elected leader, sending a full tablet report...
I20260926 22:41:38.249821 23402 raft_consensus.cc:482] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:41:38.249997 23402 raft_consensus.cc:504] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f020c35b1c3457589c04c04d6a72165" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45825 } } peers { permanent_uuid: "632318d2043c42b097640fc9786a882e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34379 } } peers { permanent_uuid: "de68f210f8354027bc7108276f65c973" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 41139 } }
I20260926 22:41:38.250437 23402 leader_election.cc:288] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f020c35b1c3457589c04c04d6a72165 (127.11.107.193:45825), de68f210f8354027bc7108276f65c973 (127.11.107.195:41139)
I20260926 22:41:38.254347 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.250568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:45825 (local address 127.11.107.194:57801)
0926 22:41:38.250678 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 22:41:38.250690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:41:38.250733 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:41:38.251003 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:41:38.251005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:41:38.251006 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:41:38.251075 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:38.251076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.252020 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.252021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:38.253213 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:38.253216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.254103 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.254104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:38.254182 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.254186 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:41:38.254235 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:41:38.254276 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 22:41:38.254626 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.250831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:57801 (local address 127.11.107.193:45825)
0926 22:41:38.250874 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:41:38.250875 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:41:38.250886 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:41:38.250916 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:41:38.250917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:41:38.250926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:41:38.250996 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:41:38.251156 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.251904 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.253358 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.253985 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.254060 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.254070 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 22:41:38.254523 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 22:41:38.254564 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:41:38.254892 23063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2168c1342dc44c3daebe42c421587648" candidate_uuid: "632318d2043c42b097640fc9786a882e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f020c35b1c3457589c04c04d6a72165" is_pre_election: true
I20260926 22:41:38.255061 23063 raft_consensus.cc:2454] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 632318d2043c42b097640fc9786a882e in term 0.
I20260926 22:41:38.255311 23137 leader_election.cc:302] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f020c35b1c3457589c04c04d6a72165, 632318d2043c42b097640fc9786a882e; no voters:
I20260926 22:41:38.255441 23402 raft_consensus.cc:2781] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:41:38.255491 23402 raft_consensus.cc:482] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:41:38.255520 23402 raft_consensus.cc:3037] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:41:38.255643 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.250778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:55725 (local address 127.11.107.195:41139)
0926 22:41:38.250820 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:41:38.250822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:41:38.252158 (+ 1336us) server_negotiation.cc:407] Connection header received
0926 22:41:38.252276 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:41:38.252278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:41:38.252290 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:41:38.252320 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:41:38.252533 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.253276 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.254746 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.255278 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.255322 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.255328 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 22:41:38.255541 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 22:41:38.255575 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:41:38.255899 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.250588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:41139 (local address 127.11.107.194:55725)
0926 22:41:38.252073 (+ 1485us) negotiation.cc:107] Waiting for socket to connect
0926 22:41:38.252087 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 22:41:38.252187 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:41:38.252384 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:41:38.252385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:41:38.252387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:41:38.252463 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:38.252464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.253471 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.253472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:38.254634 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:41:38.254636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.255402 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.255403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:41:38.255468 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.255472 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:41:38.255770 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 22:41:38.255810 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1465,"thread_start_us":101,"threads_started":1}
I20260926 22:41:38.256171 23328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2168c1342dc44c3daebe42c421587648" candidate_uuid: "632318d2043c42b097640fc9786a882e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de68f210f8354027bc7108276f65c973" is_pre_election: true
I20260926 22:41:38.256315 23328 raft_consensus.cc:2454] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 632318d2043c42b097640fc9786a882e in term 0.
I20260926 22:41:38.256458 23402 raft_consensus.cc:504] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f020c35b1c3457589c04c04d6a72165" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45825 } } peers { permanent_uuid: "632318d2043c42b097640fc9786a882e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34379 } } peers { permanent_uuid: "de68f210f8354027bc7108276f65c973" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 41139 } }
I20260926 22:41:38.256603 23402 leader_election.cc:288] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [CANDIDATE]: Term 1 election: Requested vote from peers 0f020c35b1c3457589c04c04d6a72165 (127.11.107.193:45825), de68f210f8354027bc7108276f65c973 (127.11.107.195:41139)
I20260926 22:41:38.256764 23328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2168c1342dc44c3daebe42c421587648" candidate_uuid: "632318d2043c42b097640fc9786a882e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de68f210f8354027bc7108276f65c973"
I20260926 22:41:38.256790 23063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2168c1342dc44c3daebe42c421587648" candidate_uuid: "632318d2043c42b097640fc9786a882e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f020c35b1c3457589c04c04d6a72165"
I20260926 22:41:38.256841 23328 raft_consensus.cc:3037] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:41:38.256860 23063 raft_consensus.cc:3037] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:41:38.257642 23328 raft_consensus.cc:2454] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 632318d2043c42b097640fc9786a882e in term 1.
I20260926 22:41:38.257750 23063 raft_consensus.cc:2454] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 632318d2043c42b097640fc9786a882e in term 1.
I20260926 22:41:38.257910 23134 leader_election.cc:302] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 632318d2043c42b097640fc9786a882e, de68f210f8354027bc7108276f65c973; no voters:
I20260926 22:41:38.258009 23402 raft_consensus.cc:2781] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:41:38.258150 23402 raft_consensus.cc:686] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [term 1 LEADER]: Becoming Leader. State: Replica: 632318d2043c42b097640fc9786a882e, State: Running, Role: LEADER
I20260926 22:41:38.258242 23402 consensus_queue.cc:237] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f020c35b1c3457589c04c04d6a72165" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45825 } } peers { permanent_uuid: "632318d2043c42b097640fc9786a882e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34379 } } peers { permanent_uuid: "de68f210f8354027bc7108276f65c973" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 41139 } }
I20260926 22:41:38.259058 22898 catalog_manager.cc:5495] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e reported cstate change: term changed from 0 to 1, leader changed from <none> to 632318d2043c42b097640fc9786a882e (127.11.107.194). New cstate: current_term: 1 leader_uuid: "632318d2043c42b097640fc9786a882e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f020c35b1c3457589c04c04d6a72165" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "632318d2043c42b097640fc9786a882e" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de68f210f8354027bc7108276f65c973" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 41139 } health_report { overall_health: UNKNOWN } } }
I20260926 22:41:38.287977 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.284726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.11.107.194:34379)
0926 22:41:38.284780 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 22:41:38.284781 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:41:38.284790 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:41:38.284859 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:41:38.284859 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:41:38.284879 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:41:38.284950 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:41:38.285145 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.285907 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.286979 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:41:38.287277 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:41:38.287335 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:41:38.287524 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:41:38.287648 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:41:38.287729 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 22:41:38.287883 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 22:41:38.287901 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 22:41:38.289825 23328 raft_consensus.cc:1261] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973 [term 1 FOLLOWER]: Refusing update from remote peer 632318d2043c42b097640fc9786a882e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:41:38.289886 23063 raft_consensus.cc:1261] T 2168c1342dc44c3daebe42c421587648 P 0f020c35b1c3457589c04c04d6a72165 [term 1 FOLLOWER]: Refusing update from remote peer 632318d2043c42b097640fc9786a882e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:41:38.290205 23410 consensus_queue.cc:1035] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f020c35b1c3457589c04c04d6a72165" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:41:38.290319 23402 consensus_queue.cc:1035] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de68f210f8354027bc7108276f65c973" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 41139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:41:38.295674 23414 mvcc.cc:205] Tried to move back new op lower bound from 7333734392992366592 to 7333734392866275328. Current Snapshot: MvccSnapshot[applied={T|T < 7333734392992366592}]
I20260926 22:41:38.296058 23415 mvcc.cc:205] Tried to move back new op lower bound from 7333734392992366592 to 7333734392866275328. Current Snapshot: MvccSnapshot[applied={T|T < 7333734392992366592}]
I20260926 22:41:38.299382 22897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:41:38.299470 22897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:41:38.300254 22897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e6f6140b09247f58bcfd523ba27d559: Sending DeleteTablet for 3 replicas of tablet 2168c1342dc44c3daebe42c421587648
I20260926 22:41:38.300828 23043 tablet_service.cc:1514] Processing DeleteTablet for tablet 2168c1342dc44c3daebe42c421587648 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:41:38 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:58358
I20260926 22:41:38.301086 23181 tablet_service.cc:1514] Processing DeleteTablet for tablet 2168c1342dc44c3daebe42c421587648 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:41:38 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:53486
I20260926 22:41:38.301270 23298 tablet_service.cc:1514] Processing DeleteTablet for tablet 2168c1342dc44c3daebe42c421587648 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:41:38 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:50758
I20260926 22:41:38.301662 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 22979
I20260926 22:41:38.321291 23423 tablet_replica.cc:330] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e: stopping tablet replica
I20260926 22:41:38.321508 23424 tablet_replica.cc:330] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973: stopping tablet replica
I20260926 22:41:38.321727 23424 raft_consensus.cc:2229] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 22:41:38.322105 22885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:45825 (error 108)
Sep 26 22:41:38 dist-test-slave-j8sq krb5kdc[22422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462497, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:41:38.322594 23423 raft_consensus.cc:2229] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:41:38.322811 23424 raft_consensus.cc:2258] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:41:38.322992 23423 raft_consensus.cc:2258] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:41:38.323604 23423 ts_tablet_manager.cc:1905] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:41:38.324075 23424 ts_tablet_manager.cc:1905] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:41:38.327126 23424 ts_tablet_manager.cc:1918] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:41:38.327265 23424 log.cc:1196] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/2168c1342dc44c3daebe42c421587648
I20260926 22:41:38.327992 23424 ts_tablet_manager.cc:1939] T 2168c1342dc44c3daebe42c421587648 P de68f210f8354027bc7108276f65c973: Deleting consensus metadata
I20260926 22:41:38.328506 22882 catalog_manager.cc:4841] TS de68f210f8354027bc7108276f65c973 (127.11.107.195:41139): tablet 2168c1342dc44c3daebe42c421587648 (table test-table [id=73339b6e38e1423593de8f51e965d8b1]) successfully deleted
I20260926 22:41:38.328962 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:41:38.328852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:45825 (local address 127.0.0.1:58366)
0926 22:41:38.328900 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 22:41:38.328931 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:45825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 22:41:38.329150 22885 catalog_manager.cc:4584] TS 0f020c35b1c3457589c04c04d6a72165 (127.11.107.193:45825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2168c1342dc44c3daebe42c421587648: Network error: Client connection negotiation failed: client connection to 127.11.107.193:45825: connect: Connection refused (error 111)
I20260926 22:41:38.329507 23423 ts_tablet_manager.cc:1918] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:41:38.329612 23423 log.cc:1196] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/2168c1342dc44c3daebe42c421587648
I20260926 22:41:38.329955 23423 ts_tablet_manager.cc:1939] T 2168c1342dc44c3daebe42c421587648 P 632318d2043c42b097640fc9786a882e: Deleting consensus metadata
I20260926 22:41:38.330390 22882 catalog_manager.cc:4841] TS 632318d2043c42b097640fc9786a882e (127.11.107.194:34379): tablet 2168c1342dc44c3daebe42c421587648 (table test-table [id=73339b6e38e1423593de8f51e965d8b1]) successfully deleted
I20260926 22:41:38.330639 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 23116
I20260926 22:41:38.337286 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 23253
I20260926 22:41:38.346249 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 22866
I20260926 22:41:38.353307 11695 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 22:41:38.697630 11695 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 22:41:38.697727 11695 mini_ranger.cc:66] Stopping Ranger...
I20260926 22:41:39.041953 11695 mini_ranger.cc:68] Stopped Ranger
2026-09-26 22:41:39.042 UTC [22452] LOG: received smart shutdown request
2026-09-26 22:41:39.045 UTC [22452] LOG: background worker "logical replication launcher" (PID 22461) exited with exit code 1
2026-09-26 22:41:39.045 UTC [22456] LOG: shutting down
2026-09-26 22:41:39.206 UTC [22452] LOG: database system is shut down
2026-09-26T22:41:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44876 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:41:39 dist-test-slave-j8sq krb5kdc[23434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:41:39 dist-test-slave-j8sq krb5kdc[23434](info): set up 2 sockets
Sep 26 22:41:39 dist-test-slave-j8sq krb5kdc[23434](info): commencing operation
krb5kdc: starting...
W20260926 22:41:40.354825 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:41:40 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:41:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:41:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.107.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.107.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/rangeradmin_127.11.107.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/rangeradmin_127.11.107.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.107.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.107.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/rangerlookup_127.11.107.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/rangerlookup_127.11.107.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.107.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.107.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/HTTP_127.11.107.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/HTTP_127.11.107.212@KRBTEST.COM.keytab.
I20260926 22:41:40.507827 11695 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4c2jnj/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/postgres -l logfile start
2026-09-26 22:41:42.896 UTC [23464] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 22:41:42.896 UTC [23464] LOG: listening on IPv4 address "127.11.107.212", port 35791
2026-09-26 22:41:42.900 UTC [23464] LOG: listening on Unix socket "/tmp/.s.PGSQL.35791"
2026-09-26 22:41:42.920 UTC [23467] LOG: database system was shut down at 2026-09-26 22:41:41 UTC
2026-09-26 22:41:42.923 UTC [23464] LOG: database system is ready to accept connections
I20260926 22:41:43.875240 11695 mini_postgres.cc:96] Postgres bound to 35791
2026-09-26 22:41:43.881 UTC [23475] FATAL: database "slave" does not exist
127.11.107.212:35791 - accepting connections
I20260926 22:41:43.881556 11695 mini_ranger.cc:161] Starting Ranger...
I20260926 22:41:43.896361 11695 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 22:41:44.346310 11695 mini_ranger.cc:87] Created ranger Postgres database
I20260926 22:41:44.346395 11695 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-admin
2026-09-26 22:41:44,616 [I] DB FLAVOR :POSTGRES
2026-09-26 22:41:44,617 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 22:41:44,617 [I] Checking connection..
2026-09-26 22:41:44,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 22:41:44,839 [I] Checking connection passed.
2026-09-26 22:41:44,839 [I] --------- Verifying version history table ---------
2026-09-26 22:41:44,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 22:41:45,064 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 22:41:45,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 22:41:45,303 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 22:41:45,303 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 22:41:45,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 22:41:45.555 UTC [23539] WARNING: there is no transaction in progress
2026-09-26 22:41:45,561 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 22:41:45,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 22:41:45,818 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 22:41:45,818 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 22:41:45,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 22:41:46,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 22:41:46,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j8sq.c.gcp-upstream.internal','N') ;"
2026-09-26 22:41:46,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 22:41:46,760 [I] Table x_portal_user does not exist in database ranger
2026-09-26 22:41:46,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 22:41:46,993 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 22:41:46,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 22:41:47,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 22:41:47,483 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 22:41:47,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 22:41:48.393 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.409 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.425 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.441 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.458 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.474 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.527 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.533 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.547 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.558 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.568 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.581 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.590 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.596 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.602 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:48.608 UTC [23647] WARNING: there is no transaction in progress
2026-09-26 22:41:49,184 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 22:41:49,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 22:41:49,413 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 22:41:49,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 22:41:49,661 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 22:41:49,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 22:41:49,884 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 22:41:49,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 22:41:50,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 22:41:50,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j8sq.c.gcp-upstream.internal';"
2026-09-26 22:41:50,612 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 22:41:50,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 22:41:50,857 [I] DB_PATCHES have already been applied
I20260926 22:41:50.862672 11695 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-admin:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4c2jnj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 22:41:50.862747 11695 mini_ranger.cc:193] Using host: 127.11.107.212
I20260926 22:41:50.865708 11695 mini_ranger.cc:227] Ranger admin URL: http://127.11.107.212:60155
Sep 26, 2026 10:41:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 10:41:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 10:41:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 10:41:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 10:41:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 10:41:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 10:41:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 22:41:51.984859 11695 mini_ranger.cc:160] Time spent starting Ranger: real 8.103s user 0.003s sys 0.006s
Sep 26, 2026 10:42:12 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 10:42:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 10:42:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 10:42:12 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 10:42:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 22:42:13.508814 11695 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.107.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.107.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/rangerkms_127.11.107.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.107.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/rangerkms_127.11.107.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.107.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.107.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.107.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/HTTP_127.11.107.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.107.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/HTTP_127.11.107.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 22:42:13 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462533, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 22:42:13.599061 11695 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 22:42:13.607721 11695 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 22:42:14.410892 11695 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 22:42:14.410974 11695 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-kms
I20260926 22:42:14.410997 11695 mini_ranger_kms.cc:227] Using postgres at 127.11.107.212:35791
2026-09-26 22:42:14,567 [I] DB FLAVOR :POSTGRES
2026-09-26 22:42:14,567 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 22:42:14,567 [I] Checking connection
2026-09-26 22:42:14,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 22:42:14,790 [I] connection success
2026-09-26 22:42:14,790 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 22:42:14,790 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 22:42:14,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 22:42:15,018 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 22:42:15,018 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 22:42:15,018 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 22:42:15,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.107.212:35791/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 22:42:15,282 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 22:42:15.719260 11695 mini_ranger_kms.cc:310] Created kms service
I20260926 22:42:15.884081 11695 mini_ranger_kms.cc:326] Created kudu user
I20260926 22:42:15.926640 11695 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 22:42:15 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462533, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:42:16.317330 11695 mini_ranger_kms.cc:384] Added ranger policy
I20260926 22:42:16.317462 11695 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-kms:/tmp/dist-test-task4c2jnj/build/release/bin/postgresql.jar:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4c2jnj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 22:42:16.317482 11695 mini_ranger_kms.cc:240] Using host: 127.11.107.212
I20260926 22:42:16.321035 11695 mini_ranger_kms.cc:276] Ranger KMS PID: 23827
I20260926 22:42:16.321116 11695 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.107.212:36449
Sep 26, 2026 10:42:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 10:42:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 10:42:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 10:42:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 10:42:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 10:42:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 22:42:18 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790462538, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.107.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:18.571637 11695 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.973s user 0.000s sys 0.140s
I20260926 22:42:18.571744 11695 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 22:42:18.571796 11695 mini_ranger_kms.cc:401] 127.11.107.212:36449/kms/v1/keys
Sep 26 22:42:18 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462538, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.107.212@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
Sep 26 22:42:19 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462538, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.107.212@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
Sep 26 22:42:21 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:42:21.309161 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:46553
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:46553
--ranger_config_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.107.212:36449/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:21.389225 23878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:21.389468 23878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:21.389508 23878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:21.390887 23878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:42:21.390965 23878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:21.390995 23878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:21.391028 23878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:42:21.391072 23878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:42:21.392724 23878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.107.212:36449/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:46553
--ranger_config_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:46553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:21.393000 23878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:21.393234 23878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:42:21.399263 23878 server_base.cc:1034] running on GCE node
W20260926 22:42:21.399261 23884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:21.399387 23883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:21.399292 23886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:21.399817 23878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:21.400148 23878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:21.401315 23878 hybrid_clock.cc:648] HybridClock initialized: now 1790462541401303 us; error 29 us; skew 500 ppm
Sep 26 22:42:21 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462541, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:21.404189 23878 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:42:21.404718 23878 webserver.cc:460] Webserver started at http://127.11.107.254:41075/ using document root <none> and password file <none>
I20260926 22:42:21.404959 23878 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:21.405025 23878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:21.405175 23878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:42:21 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462541, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:42:21.453902 23878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "59d11a314e6b4c1c891538c4a4d49553"
format_stamp: "Formatted at 2026-09-26 22:42:21 on dist-test-slave-j8sq"
server_key: "b75943617db87bbe068f98cf2a3daea9"
server_key_iv: "f329870854a44f5d43b96573c52da8a1"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:21.454378 23878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "59d11a314e6b4c1c891538c4a4d49553"
format_stamp: "Formatted at 2026-09-26 22:42:21 on dist-test-slave-j8sq"
server_key: "b75943617db87bbe068f98cf2a3daea9"
server_key_iv: "f329870854a44f5d43b96573c52da8a1"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:21.455940 23878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:42:21.467344 23893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:21.467552 23878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:21.467662 23878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "59d11a314e6b4c1c891538c4a4d49553"
format_stamp: "Formatted at 2026-09-26 22:42:21 on dist-test-slave-j8sq"
server_key: "b75943617db87bbe068f98cf2a3daea9"
server_key_iv: "f329870854a44f5d43b96573c52da8a1"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:21.467760 23878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:21.480538 23878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:21.482535 23878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:21.482698 23878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:21.487315 23878 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:46553
I20260926 22:42:21.487411 23945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:46553 every 8 connection(s)
I20260926 22:42:21.487756 23878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:42:21.488447 23946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:21.490806 23946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553: Bootstrap starting.
I20260926 22:42:21.491405 23946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:21.491701 23946 log.cc:824] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:21.492457 23946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553: No bootstrap required, opened a new log
I20260926 22:42:21.494613 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 23878
I20260926 22:42:21.494541 23946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d11a314e6b4c1c891538c4a4d49553" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 46553 } }
I20260926 22:42:21.494705 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:42:21.494702 23946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:21.494766 23946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d11a314e6b4c1c891538c4a4d49553, State: Initialized, Role: FOLLOWER
I20260926 22:42:21.494879 11695 external_mini_cluster.cc:1428] Setting key 9d73694b579251942ca5b2e500178483
I20260926 22:42:21.494922 23946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d11a314e6b4c1c891538c4a4d49553" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 46553 } }
I20260926 22:42:21.495018 23946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:42:21.495075 23946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:42:21.495153 23946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:21.495927 23946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d11a314e6b4c1c891538c4a4d49553" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 46553 } }
I20260926 22:42:21.496068 23946 leader_election.cc:302] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59d11a314e6b4c1c891538c4a4d49553; no voters:
I20260926 22:42:21.496250 23946 leader_election.cc:288] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:42:21.496281 23951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:42:21.496479 23951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 1 LEADER]: Becoming Leader. State: Replica: 59d11a314e6b4c1c891538c4a4d49553, State: Running, Role: LEADER
I20260926 22:42:21.496567 23946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:42:21.497162 23951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d11a314e6b4c1c891538c4a4d49553" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 46553 } }
I20260926 22:42:21.497575 23953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59d11a314e6b4c1c891538c4a4d49553. Latest consensus state: current_term: 1 leader_uuid: "59d11a314e6b4c1c891538c4a4d49553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d11a314e6b4c1c891538c4a4d49553" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 46553 } } }
I20260926 22:42:21.497646 23953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:21.497951 23946 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 22:42:21.498034 23952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59d11a314e6b4c1c891538c4a4d49553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d11a314e6b4c1c891538c4a4d49553" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 46553 } } }
I20260926 22:42:21.498096 23952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:21.500648 23965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:42:21.501802 23965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 22:42:21 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:21.506202 23965 catalog_manager.cc:1347] Generated new cluster ID: eafe747011324ec8892f43d91169e371
I20260926 22:42:21.506258 23965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:42:21.509233 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:21.495371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.11.107.254:46553)
0926 22:42:21.495542 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:21.495553 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:21.495583 (+ 30us) server_negotiation.cc:407] Connection header received
0926 22:42:21.495698 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:21.495706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:21.495761 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:21.495920 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:21.496622 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:21.498126 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:21.499647 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:21.501501 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:21.501662 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:21.504919 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:21.504923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:21.504924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:21.504935 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:21.507289 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:21.507644 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:21.507647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:21.507647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:21.507745 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:21.507913 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:21.507914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:21.507914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:21.508206 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:21.508717 (+ 511us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:21.508790 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 22:42:21.508958 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260926 22:42:21.521796 23965 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:42:21.522567 23965 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:42:21.530546 23965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553: Generated new TSK 0
I20260926 22:42:21.530825 23965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:42:22.272019 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.107.212:36449/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:46553
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:22.359823 23991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:22.360005 23991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:22.360026 23991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:22.361114 23991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:22.361152 23991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:22.361178 23991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:42:22.362705 23991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.107.212:36449/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:46553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:22.362918 23991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:22.363130 23991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:22.368932 23997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:22.369040 23999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:22.369102 23991 server_base.cc:1034] running on GCE node
W20260926 22:42:22.368942 23996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:22.369417 23991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:22.369782 23991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:22.370990 23991 hybrid_clock.cc:648] HybridClock initialized: now 1790462542370977 us; error 29 us; skew 500 ppm
Sep 26 22:42:22 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462542, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:22.374107 23991 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:42:22.374490 23991 webserver.cc:460] Webserver started at http://127.11.107.193:46407/ using document root <none> and password file <none>
I20260926 22:42:22.374742 23991 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:22.374811 23991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:22.374956 23991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:42:22 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462542, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:42:22.396474 23991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "c687831b599a402ea69e477d5cda0d86"
format_stamp: "Formatted at 2026-09-26 22:42:22 on dist-test-slave-j8sq"
server_key: "33912e693ad1d464fef23f9d30d40994"
server_key_iv: "ebe43e00835c92c0a25a059cae7d088e"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:22.397069 23991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "c687831b599a402ea69e477d5cda0d86"
format_stamp: "Formatted at 2026-09-26 22:42:22 on dist-test-slave-j8sq"
server_key: "33912e693ad1d464fef23f9d30d40994"
server_key_iv: "ebe43e00835c92c0a25a059cae7d088e"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:22.399181 23991 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260926 22:42:22.417568 24006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:22.417799 23991 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260926 22:42:22.418023 23991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "c687831b599a402ea69e477d5cda0d86"
format_stamp: "Formatted at 2026-09-26 22:42:22 on dist-test-slave-j8sq"
server_key: "33912e693ad1d464fef23f9d30d40994"
server_key_iv: "ebe43e00835c92c0a25a059cae7d088e"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:22.418213 23991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:22.441613 23991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:22.443979 23991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:22.444142 23991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:22.444412 23991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:22.444804 23991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:22.444855 23991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:22.444886 23991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:22.444904 23991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:22.451483 23991 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:34435
I20260926 22:42:22.451949 23991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:42:22.453701 24121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:34435 every 8 connection(s)
I20260926 22:42:22.458024 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 23991
I20260926 22:42:22.458127 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:42:22.458325 11695 external_mini_cluster.cc:1428] Setting key 19bb044310fbfe4ed4d815b71afe23be
Sep 26 22:42:22 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462542, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:22.465812 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:22.454324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:46553 (local address 127.11.107.193:56239)
0926 22:42:22.454511 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:22.454550 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:22.455064 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:22.455341 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:22.455345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:22.455524 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:22.455711 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:22.455715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:22.456701 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:22.456702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:22.457723 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:22.457726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:22.458305 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:22.458306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:22.458443 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:22.459283 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:22.459293 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:22.461998 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:22.464081 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:22.464087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:22.464090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:22.464396 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:22.464633 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:22.464634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:22.464634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:22.464777 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:22.465097 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:22.465103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:22.465262 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 22:42:22.465356 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 22:42:22.465518 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
I20260926 22:42:22.466396 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:22.454325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:56239 (local address 127.11.107.254:46553)
0926 22:42:22.454453 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:22.454456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:22.454663 (+ 207us) server_negotiation.cc:407] Connection header received
0926 22:42:22.455193 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:22.455195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:22.455210 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:22.455274 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:22.455809 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:22.456595 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:22.457816 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:22.458151 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:22.458221 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:22.462093 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:22.462095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:22.462095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:22.462104 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:22.463975 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:22.464487 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:22.464488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:22.464488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:22.464552 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:22.464861 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:22.464862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:22.464863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:22.465006 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:22.466117 (+ 1111us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:22.466159 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 22:42:22.466261 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260926 22:42:22.466886 24122 heartbeater.cc:347] Connected to a master server at 127.11.107.254:46553
I20260926 22:42:22.466972 24122 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:22.467139 24122 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
I20260926 22:42:22.467694 23910 ts_manager.cc:194] Registered new tserver with Master: c687831b599a402ea69e477d5cda0d86 (127.11.107.193:34435)
I20260926 22:42:22.468565 23910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:56239
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:42:22.489507 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.107.212:36449/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:46553
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:22.600931 24128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:22.601186 24128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:22.601220 24128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:22.602913 24128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:22.602973 24128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:22.603024 24128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:42:22.605147 24128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.107.212:36449/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:46553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:22.605371 24128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:22.605648 24128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 26 22:42:22 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790462542, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 22:42:22.611745 24136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:22.611989 24135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:22.612223 24128 server_base.cc:1034] running on GCE node
W20260926 22:42:22.612193 24138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:22.612538 24128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:22.612823 24128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:22.613958 24128 hybrid_clock.cc:648] HybridClock initialized: now 1790462542613937 us; error 33 us; skew 500 ppm
Sep 26 22:42:22 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462542, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:22.617208 24128 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:42:22.617753 24128 webserver.cc:460] Webserver started at http://127.11.107.194:39481/ using document root <none> and password file <none>
I20260926 22:42:22.618081 24128 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:22.618201 24128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:22.618377 24128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:42:22 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462542, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:42:22.641819 24128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "d406962f79334ed0bbde8c3facb3c15b"
format_stamp: "Formatted at 2026-09-26 22:42:22 on dist-test-slave-j8sq"
server_key: "1c28e0246e72d20f8ac1f63f5ebbb204"
server_key_iv: "2c45fe1111db0ddb95aa814e65d30a4b"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:22.642390 24128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "d406962f79334ed0bbde8c3facb3c15b"
format_stamp: "Formatted at 2026-09-26 22:42:22 on dist-test-slave-j8sq"
server_key: "1c28e0246e72d20f8ac1f63f5ebbb204"
server_key_iv: "2c45fe1111db0ddb95aa814e65d30a4b"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:22.646688 24128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20260926 22:42:22.656742 24145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:22.656952 24128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:22.657028 24128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "d406962f79334ed0bbde8c3facb3c15b"
format_stamp: "Formatted at 2026-09-26 22:42:22 on dist-test-slave-j8sq"
server_key: "1c28e0246e72d20f8ac1f63f5ebbb204"
server_key_iv: "2c45fe1111db0ddb95aa814e65d30a4b"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:22.657109 24128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:22.669471 24128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:22.671962 24128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:22.672202 24128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:22.672456 24128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:22.672821 24128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:22.672909 24128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:22.672979 24128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:22.673139 24128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:22.679864 24128 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:35905
I20260926 22:42:22.680328 24128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:42:22.681970 24258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:35905 every 8 connection(s)
I20260926 22:42:22.682519 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 24128
I20260926 22:42:22.682619 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:42:22.682818 11695 external_mini_cluster.cc:1428] Setting key 3602ca0e4458f825a0ebdc157491982e
Sep 26 22:42:22 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462542, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:22.692986 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:22.681989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:45613 (local address 127.11.107.254:46553)
0926 22:42:22.682037 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:22.682039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:22.682916 (+ 877us) server_negotiation.cc:407] Connection header received
0926 22:42:22.683608 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:22.683609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:22.683624 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:22.683655 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:22.684459 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:22.685195 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:22.686691 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:22.686983 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:22.687032 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:22.689586 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:22.689588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:22.689588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:22.689596 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:22.691354 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:22.691858 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:22.691859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:22.691859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:22.691920 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:22.692233 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:22.692234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:22.692234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:22.692385 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:22.692463 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:22.692798 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 22:42:22.692893 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:42:22.693212 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:22.682167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:46553 (local address 127.11.107.194:45613)
0926 22:42:22.682777 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:22.682826 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:22.683482 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:22.683832 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:22.683837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:22.684065 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:22.684343 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:22.684350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:22.685335 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:22.685336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:22.686587 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:22.686591 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:22.687142 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:22.687142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:22.687290 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:22.687865 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:22.687874 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:22.689483 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:22.691454 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:22.691460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:22.691463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:22.691767 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:22.691999 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:22.692000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:22.692000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:22.692147 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:22.692465 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:22.692470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:22.692633 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 22:42:22.692752 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 22:42:22.692923 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":79,"threads_started":1}
I20260926 22:42:22.693789 24259 heartbeater.cc:347] Connected to a master server at 127.11.107.254:46553
I20260926 22:42:22.693898 24259 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:22.694080 24259 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
I20260926 22:42:22.694459 23910 ts_manager.cc:194] Registered new tserver with Master: d406962f79334ed0bbde8c3facb3c15b (127.11.107.194:35905)
I20260926 22:42:22.694938 23910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:45613
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:42:22.720695 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.107.212:36449/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:46553
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:22.841746 24265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:22.842109 24265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:22.842187 24265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:22.843801 24265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:22.843888 24265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:22.843962 24265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:42:22.845971 24265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.107.212:36449/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:46553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:22.846524 24265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:22.846836 24265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:22.853173 24271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:22.853538 24273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:22.854578 24265 server_base.cc:1034] running on GCE node
W20260926 22:42:22.858016 24270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:22.858289 24265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:22.858592 24265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:22.859767 24265 hybrid_clock.cc:648] HybridClock initialized: now 1790462542859749 us; error 34 us; skew 500 ppm
Sep 26 22:42:22 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462542, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:22.863168 24265 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:42:22.863649 24265 webserver.cc:460] Webserver started at http://127.11.107.195:34207/ using document root <none> and password file <none>
I20260926 22:42:22.863881 24265 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:22.863929 24265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:22.864045 24265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:42:22 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462542, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:42:22.881003 24265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a"
format_stamp: "Formatted at 2026-09-26 22:42:22 on dist-test-slave-j8sq"
server_key: "1e7c55022b0d73d9bdb9a90c3d0e12b2"
server_key_iv: "dd11e9ecf948aec66456986b9fd8f8cd"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:22.881476 24265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a"
format_stamp: "Formatted at 2026-09-26 22:42:22 on dist-test-slave-j8sq"
server_key: "1e7c55022b0d73d9bdb9a90c3d0e12b2"
server_key_iv: "dd11e9ecf948aec66456986b9fd8f8cd"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:22.888899 24265 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.000s
I20260926 22:42:22.905974 24280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:22.906147 24265 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 22:42:22.906236 24265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a"
format_stamp: "Formatted at 2026-09-26 22:42:22 on dist-test-slave-j8sq"
server_key: "1e7c55022b0d73d9bdb9a90c3d0e12b2"
server_key_iv: "dd11e9ecf948aec66456986b9fd8f8cd"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:22.906327 24265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:22.939682 24265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:22.942101 24265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:22.942253 24265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:22.942512 24265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:22.942896 24265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:22.942946 24265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:22.942984 24265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:22.943010 24265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:22.949350 24265 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:38653
I20260926 22:42:22.949786 24265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:42:22.955508 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 24265
I20260926 22:42:22.955610 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:42:22.955804 11695 external_mini_cluster.cc:1428] Setting key 34567f28012759f39793832617243898
I20260926 22:42:22.961913 24393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:38653 every 8 connection(s)
Sep 26 22:42:22 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462542, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:22.978483 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:22.961797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:58039 (local address 127.11.107.254:46553)
0926 22:42:22.961870 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:22.961872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:22.962351 (+ 479us) server_negotiation.cc:407] Connection header received
0926 22:42:22.963426 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:22.963427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:22.963439 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:22.963470 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:22.964304 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:22.965045 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:22.966779 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:22.967089 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:22.967132 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:22.972435 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:22.972436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:22.972437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:22.972444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:22.974173 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:22.975855 (+ 1682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:22.975857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:22.975857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:22.975915 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:22.976204 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:22.976205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:22.976206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:22.976335 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:22.976390 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:22.978307 (+ 1917us) server_negotiation.cc:299] Negotiation successful
0926 22:42:22.978389 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 22:42:22.979398 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:22.961478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:46553 (local address 127.11.107.195:58039)
0926 22:42:22.962230 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:22.962272 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:22.963302 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:22.963654 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:22.963658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:22.963904 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:22.964195 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:22.964202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:22.965167 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:22.965168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:22.966682 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:22.966684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:22.967234 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:22.967235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:22.967372 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:22.968477 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:22.968487 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:22.972321 (+ 3834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:22.975450 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:22.975456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:22.975458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:22.975745 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:22.975986 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:22.975986 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:22.975986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:22.976124 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:22.976954 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:22.976960 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:22.977127 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 22:42:22.978991 (+ 1864us) client_negotiation.cc:240] Negotiation successful
0926 22:42:22.979165 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":89,"threads_started":1}
I20260926 22:42:22.980024 24394 heartbeater.cc:347] Connected to a master server at 127.11.107.254:46553
I20260926 22:42:22.980125 24394 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:22.980301 24394 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
I20260926 22:42:22.980688 23910 ts_manager.cc:194] Registered new tserver with Master: ae5ac1901e7a4d5c832927a8f5a6b92a (127.11.107.195:38653)
I20260926 22:42:22.981169 23910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:58039
I20260926 22:42:22.988775 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:42:22.989522 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 23991
I20260926 22:42:22.995222 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 24128
I20260926 22:42:23.001101 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 24265
I20260926 22:42:23.007084 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 23878
I20260926 22:42:23.016016 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:46553
--webserver_interface=127.11.107.254
--webserver_port=41075
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:46553
--ranger_config_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.107.212:36449/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.107.212:36449/kms with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:23.097175 24397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:23.097373 24397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:23.097394 24397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:23.098587 24397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:42:23.098626 24397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:23.098640 24397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:23.098654 24397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:42:23.098665 24397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:42:23.100178 24397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.107.212:36449/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:46553
--ranger_config_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:46553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=41075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:23.100404 24397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:23.100642 24397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:23.105239 24403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:23.105245 24402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:23.105479 24405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:23.105419 24397 server_base.cc:1034] running on GCE node
I20260926 22:42:23.105760 24397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:23.106096 24397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:23.107252 24397 hybrid_clock.cc:648] HybridClock initialized: now 1790462543107234 us; error 32 us; skew 500 ppm
Sep 26 22:42:23 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:23.109649 24397 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:42:23.110152 24397 webserver.cc:460] Webserver started at http://127.11.107.254:41075/ using document root <none> and password file <none>
I20260926 22:42:23.110374 24397 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:23.110422 24397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 22:42:23 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:42:23.170149 24397 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:23.171036 24413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:23.171231 24397 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:23.171343 24397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "59d11a314e6b4c1c891538c4a4d49553"
format_stamp: "Formatted at 2026-09-26 22:42:21 on dist-test-slave-j8sq"
server_key: "b75943617db87bbe068f98cf2a3daea9"
server_key_iv: "f329870854a44f5d43b96573c52da8a1"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:23.171710 24397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:23.193925 24397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:23.196022 24397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:23.196209 24397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:23.201365 24397 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:46553
I20260926 22:42:23.201450 24465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:46553 every 8 connection(s)
I20260926 22:42:23.201794 24397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:42:23.203028 24466 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:42:23.203696 24466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553: Bootstrap starting.
I20260926 22:42:23.211483 24466 log.cc:824] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:23.211993 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 24397
I20260926 22:42:23.212314 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.107.212:36449/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:34435
--local_ip_for_outbound_sockets=127.11.107.193
--tserver_master_addrs=127.11.107.254:46553
--webserver_port=46407
--webserver_interface=127.11.107.193
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.107.212:36449/kms with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:42:23.213686 24466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:42:23.214020 24466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553: Bootstrap complete.
I20260926 22:42:23.217917 24466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d11a314e6b4c1c891538c4a4d49553" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 46553 } }
I20260926 22:42:23.218313 24466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d11a314e6b4c1c891538c4a4d49553, State: Initialized, Role: FOLLOWER
I20260926 22:42:23.218483 24466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d11a314e6b4c1c891538c4a4d49553" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 46553 } }
I20260926 22:42:23.218575 24466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:42:23.218621 24466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:42:23.218659 24466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:42:23.220201 24466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d11a314e6b4c1c891538c4a4d49553" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 46553 } }
I20260926 22:42:23.220317 24466 leader_election.cc:302] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59d11a314e6b4c1c891538c4a4d49553; no voters:
I20260926 22:42:23.220486 24466 leader_election.cc:288] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 22:42:23.220561 24470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 22:42:23.220710 24470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [term 2 LEADER]: Becoming Leader. State: Replica: 59d11a314e6b4c1c891538c4a4d49553, State: Running, Role: LEADER
I20260926 22:42:23.220752 24466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:42:23.220835 24470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d11a314e6b4c1c891538c4a4d49553" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 46553 } }
I20260926 22:42:23.221081 24470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "59d11a314e6b4c1c891538c4a4d49553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d11a314e6b4c1c891538c4a4d49553" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 46553 } } }
I20260926 22:42:23.221146 24470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:23.221266 24470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59d11a314e6b4c1c891538c4a4d49553. Latest consensus state: current_term: 2 leader_uuid: "59d11a314e6b4c1c891538c4a4d49553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d11a314e6b4c1c891538c4a4d49553" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 46553 } } }
I20260926 22:42:23.221312 24470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:23.221418 24475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:42:23.221886 24475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:42:23.222009 24475 catalog_manager.cc:1259] Loaded cluster ID: eafe747011324ec8892f43d91169e371
I20260926 22:42:23.222044 24475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:42:23.222296 24466 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 22:42:23.226071 24475 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:42:23.226629 24475 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553: Loaded TSK: 0
I20260926 22:42:23.226873 24475 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 22:42:23.314042 24469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:23.314397 24469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:23.314494 24469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:23.315814 24469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:23.315882 24469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:23.315924 24469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:42:23.317370 24469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.107.212:36449/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:34435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=46407
--tserver_master_addrs=127.11.107.254:46553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:23.317625 24469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:23.317890 24469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:23.322784 24504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:23.323029 24506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:23.323344 24503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:23.323573 24469 server_base.cc:1034] running on GCE node
I20260926 22:42:23.323769 24469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:23.324002 24469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:23.325171 24469 hybrid_clock.cc:648] HybridClock initialized: now 1790462543325150 us; error 36 us; skew 500 ppm
Sep 26 22:42:23 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:23.330632 24469 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:42:23.331115 24469 webserver.cc:460] Webserver started at http://127.11.107.193:46407/ using document root <none> and password file <none>
I20260926 22:42:23.331321 24469 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:23.331367 24469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 22:42:23 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:42:23.356635 24469 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260926 22:42:23.358059 24514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:23.358238 24469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:23.358328 24469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "c687831b599a402ea69e477d5cda0d86"
format_stamp: "Formatted at 2026-09-26 22:42:22 on dist-test-slave-j8sq"
server_key: "33912e693ad1d464fef23f9d30d40994"
server_key_iv: "ebe43e00835c92c0a25a059cae7d088e"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:23.358798 24469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:23.379245 24469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:23.381598 24469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:23.381764 24469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:23.382035 24469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:23.382385 24469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:23.382431 24469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:23.382467 24469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:23.382493 24469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:23.389005 24469 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:34435
I20260926 22:42:23.389444 24469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:42:23.392575 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 24469
I20260926 22:42:23.392848 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.107.212:36449/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:35905
--local_ip_for_outbound_sockets=127.11.107.194
--tserver_master_addrs=127.11.107.254:46553
--webserver_port=39481
--webserver_interface=127.11.107.194
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.107.212:36449/kms with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:42:23.394726 24627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:34435 every 8 connection(s)
Sep 26 22:42:23 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:23.413754 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:23.401384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:46553 (local address 127.11.107.193:33973)
0926 22:42:23.401648 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:23.401694 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:23.402506 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:23.403276 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:23.403280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:23.403500 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:23.403753 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:23.403758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.405380 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.405381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:23.406692 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:23.406694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.407589 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.407590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:23.407711 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:23.408312 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:23.408322 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:23.409744 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:23.411965 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:23.411971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:23.411973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:23.412300 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:23.412584 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:23.412585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:23.412585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:23.412703 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:23.413131 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:23.413137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:23.413287 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 22:42:23.413391 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 22:42:23.413532 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
I20260926 22:42:23.414673 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:23.401434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:33973 (local address 127.11.107.254:46553)
0926 22:42:23.402728 (+ 1294us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:23.402739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:23.402774 (+ 35us) server_negotiation.cc:407] Connection header received
0926 22:42:23.402926 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:23.402934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:23.402993 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:23.403119 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:23.403874 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.405265 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.406796 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.407392 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.407483 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:23.409898 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:23.409902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:23.409903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:23.409914 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:23.411873 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:23.412401 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:23.412404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:23.412404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:23.412502 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:23.412772 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:23.412773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:23.412773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:23.413033 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:23.413102 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:23.414325 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0926 22:42:23.414450 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1235,"thread_start_us":65,"threads_started":1}
I20260926 22:42:23.418243 24628 heartbeater.cc:347] Connected to a master server at 127.11.107.254:46553
I20260926 22:42:23.418345 24628 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:23.418536 24628 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
W20260926 22:42:23.419348 24628 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:46553 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 22:42:23.419411 24628 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:23.419468 24628 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
I20260926 22:42:23.419833 24628 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:23.419898 24628 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
W20260926 22:42:23.420200 24628 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 22:42:23.495285 24630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:23.495550 24630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:23.495591 24630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:23.496937 24630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:23.497006 24630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:23.497052 24630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:42:23.498744 24630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.107.212:36449/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:35905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=39481
--tserver_master_addrs=127.11.107.254:46553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:23.498975 24630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:23.499235 24630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:23.505928 24638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:23.505985 24640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:23.507162 24637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:23.507674 24630 server_base.cc:1034] running on GCE node
I20260926 22:42:23.507882 24630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:23.508162 24630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:23.509327 24630 hybrid_clock.cc:648] HybridClock initialized: now 1790462543509306 us; error 38 us; skew 500 ppm
Sep 26 22:42:23 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:23.512303 24630 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:42:23.512825 24630 webserver.cc:460] Webserver started at http://127.11.107.194:39481/ using document root <none> and password file <none>
I20260926 22:42:23.513062 24630 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:23.513149 24630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 22:42:23 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:42:23.536017 24630 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 22:42:23.537222 24648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:23.537422 24630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 22:42:23.537509 24630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "d406962f79334ed0bbde8c3facb3c15b"
format_stamp: "Formatted at 2026-09-26 22:42:22 on dist-test-slave-j8sq"
server_key: "1c28e0246e72d20f8ac1f63f5ebbb204"
server_key_iv: "2c45fe1111db0ddb95aa814e65d30a4b"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:23.538012 24630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:23.551818 24630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:23.553956 24630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:23.554105 24630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:23.554337 24630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:23.554706 24630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:23.554759 24630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:23.554798 24630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:23.554829 24630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:23.561349 24630 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:35905
I20260926 22:42:23.561473 24761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:35905 every 8 connection(s)
I20260926 22:42:23.561780 24630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:42:23 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:23.571499 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:23.562629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:58583 (local address 127.11.107.254:46553)
0926 22:42:23.562676 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:23.562678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:23.562942 (+ 264us) server_negotiation.cc:407] Connection header received
0926 22:42:23.563609 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:23.563610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:23.563626 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:23.563696 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:23.564393 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.564979 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.566111 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.566343 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.566422 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:23.568493 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:23.568495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:23.568495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:23.568503 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:23.570149 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:23.570542 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:23.570543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:23.570544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:23.570600 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:23.570831 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:23.570832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:23.570832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:23.570949 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:23.571025 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:23.571323 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 22:42:23.571407 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:42:23.571658 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:23.562641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:46553 (local address 127.11.107.194:58583)
0926 22:42:23.562802 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:23.562844 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:23.563484 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:23.563773 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:23.563777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:23.563961 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:23.564284 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:23.564291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.565081 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.565082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:23.566027 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:23.566029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.566471 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.566473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:23.566596 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:23.567155 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:23.567164 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:23.568399 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:23.570228 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:23.570235 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:23.570238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:23.570453 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:23.570652 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:23.570652 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:23.570653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:23.570761 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:23.571017 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:23.571023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:23.571162 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 22:42:23.571282 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 22:42:23.571421 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260926 22:42:23.571877 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 24630
I20260926 22:42:23.572165 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.107.212:36449/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:38653
--local_ip_for_outbound_sockets=127.11.107.195
--tserver_master_addrs=127.11.107.254:46553
--webserver_port=34207
--webserver_interface=127.11.107.195
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.107.212:36449/kms with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:42:23.572494 24762 heartbeater.cc:347] Connected to a master server at 127.11.107.254:46553
I20260926 22:42:23.572608 24762 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:23.572817 24762 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
W20260926 22:42:23.573442 24762 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:46553 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 22:42:23.573537 24762 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:23.573613 24762 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
I20260926 22:42:23.573987 24762 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:23.574064 24762 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
W20260926 22:42:23.574339 24762 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 22:42:23.668609 24765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:23.668869 24765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:23.668902 24765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:23.670543 24765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:23.670590 24765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:23.670626 24765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:42:23.672878 24765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.107.212:36449/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:38653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=34207
--tserver_master_addrs=127.11.107.254:46553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:23.673105 24765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:23.673344 24765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:23.682992 24773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:23.683243 24771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:23.684996 24770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:23.687071 24765 server_base.cc:1034] running on GCE node
I20260926 22:42:23.687386 24765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:23.688010 24765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:23.689234 24765 hybrid_clock.cc:648] HybridClock initialized: now 1790462543689217 us; error 28 us; skew 500 ppm
Sep 26 22:42:23 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:23.692938 24765 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:42:23.693612 24765 webserver.cc:460] Webserver started at http://127.11.107.195:34207/ using document root <none> and password file <none>
I20260926 22:42:23.694007 24765 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:23.694142 24765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 22:42:23 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:42:23.708637 24765 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:23.714287 24781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:23.714473 24765 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260926 22:42:23.714545 24765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a"
format_stamp: "Formatted at 2026-09-26 22:42:22 on dist-test-slave-j8sq"
server_key: "1e7c55022b0d73d9bdb9a90c3d0e12b2"
server_key_iv: "dd11e9ecf948aec66456986b9fd8f8cd"
server_key_version: "kuduclusterkey@0"
I20260926 22:42:23.714967 24765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:23.749155 24765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:23.751497 24765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:23.751649 24765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:23.751873 24765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:23.752206 24765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:23.752249 24765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:23.752296 24765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:23.752321 24765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:23.759685 24765 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:38653
I20260926 22:42:23.759897 24895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:38653 every 8 connection(s)
I20260926 22:42:23.760149 24765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:42:23 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462543, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:23.769050 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:23.760935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:39087 (local address 127.11.107.254:46553)
0926 22:42:23.760981 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:23.760983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:23.761230 (+ 247us) server_negotiation.cc:407] Connection header received
0926 22:42:23.761759 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:23.761760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:23.761772 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:23.761824 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:23.762376 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.763001 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.764143 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.764359 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.764441 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:23.766454 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:23.766455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:23.766455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:23.766463 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:23.767740 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:23.768124 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:23.768125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:23.768125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:23.768190 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:23.768431 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:23.768432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:23.768433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:23.768545 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:23.768618 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:23.768885 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 22:42:23.768961 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:42:23.769419 24898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:23.760927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:46553 (local address 127.11.107.195:39087)
0926 22:42:23.761091 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:23.761126 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:23.761643 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:23.761920 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:23.761924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:23.762094 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:23.762280 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:23.762284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.763091 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.763092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:23.764060 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:23.764062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.764468 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.764469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:23.764595 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:23.765032 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:23.765040 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:23.766362 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:23.767831 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:23.767837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:23.767839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:23.768049 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:23.768255 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:23.768256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:23.768256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:23.768364 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:23.768609 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:23.768615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:23.768756 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 22:42:23.768865 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 22:42:23.768999 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260926 22:42:23.769943 24896 heartbeater.cc:347] Connected to a master server at 127.11.107.254:46553
I20260926 22:42:23.770047 24896 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:23.770083 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 24765
I20260926 22:42:23.770241 24896 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
W20260926 22:42:23.770888 24896 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:46553 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 22:42:23.770939 24896 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:23.770979 24896 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
I20260926 22:42:23.771452 24896 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:23.771524 24896 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
W20260926 22:42:23.771831 24896 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 22:42:23.783497 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:23.770466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.11.107.254:46553)
0926 22:42:23.770503 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:23.770505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:23.770781 (+ 276us) server_negotiation.cc:407] Connection header received
0926 22:42:23.771159 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:23.771161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:23.771171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:23.771220 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:23.772193 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.772768 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.773942 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:23.774233 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:23.774305 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:23.775728 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:23.775730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:23.775730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:23.775737 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:23.782392 (+ 6655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:23.782741 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:23.782742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:23.782743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:23.782794 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:23.783002 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:23.783003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:23.783003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:23.783112 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:23.783292 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:23.783327 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 22:42:23.783394 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 26 22:42:24 dist-test-slave-j8sq krb5kdc[23434](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790462544, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:24.420377 24628 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:24.420536 24628 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
I20260926 22:42:24.420997 24427 ts_manager.cc:194] Registered new tserver with Master: c687831b599a402ea69e477d5cda0d86 (127.11.107.193:34435)
I20260926 22:42:24.422144 24427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:33973
I20260926 22:42:24.574519 24762 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:24.574659 24762 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
I20260926 22:42:24.574944 24427 ts_manager.cc:194] Registered new tserver with Master: d406962f79334ed0bbde8c3facb3c15b (127.11.107.194:35905)
I20260926 22:42:24.575489 24427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:58583
Sep 26 22:42:24 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462544, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:42:24.772011 24896 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:24.772161 24896 heartbeater.cc:511] Master 127.11.107.254:46553 requested a full tablet report, sending...
I20260926 22:42:24.772557 24427 ts_manager.cc:194] Registered new tserver with Master: ae5ac1901e7a4d5c832927a8f5a6b92a (127.11.107.195:38653)
I20260926 22:42:24.773104 24427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:39087
I20260926 22:42:24.773769 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:42:24.782644 24916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:24.774694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.11.107.254:46553)
0926 22:42:24.775048 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:24.775052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:24.775059 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:42:24.775080 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:24.775081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:24.775093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:24.775842 (+ 749us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:24.775878 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.776616 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.777621 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.777973 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.778045 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:24.779458 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:24.779460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:24.779460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:24.779467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:24.781213 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:24.781615 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:24.781617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:24.781617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:24.781692 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:24.781947 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:24.781948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:24.781948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:24.782087 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:24.782197 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:24.782353 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 22:42:24.782442 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":60,"threads_started":1}
I20260926 22:42:24.784847 24427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:42:24.787200 24427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:42:24.797276 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:24.790096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35905 (local address 127.0.0.1:59136)
0926 22:42:24.790241 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:24.790309 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:24.790401 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:24.791318 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:24.791321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:24.791324 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:24.791386 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:24.791388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.792969 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.792970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:24.794231 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:24.794235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.797005 (+ 2770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.797006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:24.797088 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:24.797104 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 22:42:24.797153 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:42:24.797201 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260926 22:42:24.797933 24924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:24.790396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:38653 (local address 127.0.0.1:46164)
0926 22:42:24.793940 (+ 3544us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:24.793951 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:24.793998 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:24.794327 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:24.794328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:24.794330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:24.794382 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:24.794384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.795539 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.795539 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:24.796594 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:24.796597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.797714 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.797716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:24.797771 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:24.797775 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:42:24.797802 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 22:42:24.797869 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3528,"thread_start_us":52,"threads_started":1}
I20260926 22:42:24.798020 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:24.790726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.11.107.194:35905)
0926 22:42:24.791148 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:24.791153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:24.791162 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:42:24.791198 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:24.791200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:24.791216 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:24.791465 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:24.791544 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.792589 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.794337 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.796904 (+ 2567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.797501 (+ 597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:24.797565 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:24.797591 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 22:42:24.797637 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":55,"threads_started":1}
I20260926 22:42:24.798537 24926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:24.790556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46164 (local address 127.11.107.195:38653)
0926 22:42:24.794113 (+ 3557us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:24.794118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:24.794127 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:42:24.794160 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:24.794162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:24.794177 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:24.794243 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:24.794446 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.795439 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.796696 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.797604 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.798336 (+ 732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:24.798398 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:24.798422 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 22:42:24.798463 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3524,"thread_start_us":53,"threads_started":1}
I20260926 22:42:24.798709 24691 tablet_service.cc:1467] Processing CreateTablet for tablet 058e68b127ef4b35af18409f0b1ed973 (DEFAULT_TABLE table=test-table [id=bc002145233446c0acff4521ee57d3c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:42:24.799072 24691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058e68b127ef4b35af18409f0b1ed973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:24.799305 24830 tablet_service.cc:1467] Processing CreateTablet for tablet 058e68b127ef4b35af18409f0b1ed973 (DEFAULT_TABLE table=test-table [id=bc002145233446c0acff4521ee57d3c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:42:24.799631 24830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058e68b127ef4b35af18409f0b1ed973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:24.800535 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:24.790283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:34435 (local address 127.0.0.1:40524)
0926 22:42:24.790782 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:24.790792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:24.790840 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:24.795456 (+ 4616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:24.795457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:24.795459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:24.795513 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:24.795514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.798329 (+ 2815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.798330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:24.799362 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:24.799365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.800334 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.800335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:24.800403 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:24.800406 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:42:24.800434 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 22:42:24.800473 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":51,"threads_started":1}
I20260926 22:42:24.800923 24927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:24.790641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.11.107.193:34435)
0926 22:42:24.795299 (+ 4658us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:24.795303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:24.795313 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:42:24.795345 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:24.795347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:24.795362 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:24.795577 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:24.795646 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.796650 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.799466 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.800224 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.800716 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:24.800777 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:24.800802 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 22:42:24.800844 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4622,"thread_start_us":52,"threads_started":1}
I20260926 22:42:24.801575 24562 tablet_service.cc:1467] Processing CreateTablet for tablet 058e68b127ef4b35af18409f0b1ed973 (DEFAULT_TABLE table=test-table [id=bc002145233446c0acff4521ee57d3c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:42:24.801909 24562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058e68b127ef4b35af18409f0b1ed973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:24.806015 24930 tablet_bootstrap.cc:492] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a: Bootstrap starting.
I20260926 22:42:24.806701 24930 tablet_bootstrap.cc:654] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:24.807018 24930 log.cc:824] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:24.807343 24931 tablet_bootstrap.cc:492] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86: Bootstrap starting.
I20260926 22:42:24.807935 24931 tablet_bootstrap.cc:654] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:24.808223 24931 log.cc:824] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:24.808519 24929 tablet_bootstrap.cc:492] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b: Bootstrap starting.
I20260926 22:42:24.809170 24929 tablet_bootstrap.cc:654] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:24.809561 24929 log.cc:824] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:24.810300 24929 tablet_bootstrap.cc:492] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b: No bootstrap required, opened a new log
I20260926 22:42:24.810372 24929 ts_tablet_manager.cc:1397] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:42:24.812604 24929 raft_consensus.cc:348] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406962f79334ed0bbde8c3facb3c15b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35905 } } peers { permanent_uuid: "c687831b599a402ea69e477d5cda0d86" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34435 } } peers { permanent_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38653 } }
I20260926 22:42:24.812791 24929 raft_consensus.cc:374] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:24.812834 24929 raft_consensus.cc:729] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d406962f79334ed0bbde8c3facb3c15b, State: Initialized, Role: FOLLOWER
I20260926 22:42:24.812953 24929 consensus_queue.cc:260] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406962f79334ed0bbde8c3facb3c15b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35905 } } peers { permanent_uuid: "c687831b599a402ea69e477d5cda0d86" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34435 } } peers { permanent_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38653 } }
I20260926 22:42:24.813207 24929 ts_tablet_manager.cc:1428] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:42:24.813385 24762 heartbeater.cc:503] Master 127.11.107.254:46553 was elected leader, sending a full tablet report...
I20260926 22:42:24.813774 24930 tablet_bootstrap.cc:492] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a: No bootstrap required, opened a new log
I20260926 22:42:24.813881 24930 ts_tablet_manager.cc:1397] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260926 22:42:24.816360 24930 raft_consensus.cc:348] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406962f79334ed0bbde8c3facb3c15b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35905 } } peers { permanent_uuid: "c687831b599a402ea69e477d5cda0d86" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34435 } } peers { permanent_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38653 } }
I20260926 22:42:24.816524 24930 raft_consensus.cc:374] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:24.816565 24930 raft_consensus.cc:729] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5ac1901e7a4d5c832927a8f5a6b92a, State: Initialized, Role: FOLLOWER
I20260926 22:42:24.816711 24930 consensus_queue.cc:260] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406962f79334ed0bbde8c3facb3c15b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35905 } } peers { permanent_uuid: "c687831b599a402ea69e477d5cda0d86" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34435 } } peers { permanent_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38653 } }
I20260926 22:42:24.816957 24930 ts_tablet_manager.cc:1428] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:42:24.817077 24896 heartbeater.cc:503] Master 127.11.107.254:46553 was elected leader, sending a full tablet report...
I20260926 22:42:24.817593 24931 tablet_bootstrap.cc:492] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86: No bootstrap required, opened a new log
I20260926 22:42:24.817672 24931 ts_tablet_manager.cc:1397] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260926 22:42:24.819774 24931 raft_consensus.cc:348] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406962f79334ed0bbde8c3facb3c15b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35905 } } peers { permanent_uuid: "c687831b599a402ea69e477d5cda0d86" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34435 } } peers { permanent_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38653 } }
I20260926 22:42:24.819957 24931 raft_consensus.cc:374] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:24.819999 24931 raft_consensus.cc:729] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c687831b599a402ea69e477d5cda0d86, State: Initialized, Role: FOLLOWER
I20260926 22:42:24.820119 24931 consensus_queue.cc:260] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406962f79334ed0bbde8c3facb3c15b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35905 } } peers { permanent_uuid: "c687831b599a402ea69e477d5cda0d86" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34435 } } peers { permanent_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38653 } }
I20260926 22:42:24.820381 24628 heartbeater.cc:503] Master 127.11.107.254:46553 was elected leader, sending a full tablet report...
I20260926 22:42:24.820371 24931 ts_tablet_manager.cc:1428] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
Sep 26 22:42:24 dist-test-slave-j8sq krb5kdc[23434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462544, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for HTTP/127.11.107.212@KRBTEST.COM
I20260926 22:42:24.865144 24936 raft_consensus.cc:482] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:42:24.865299 24936 raft_consensus.cc:504] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406962f79334ed0bbde8c3facb3c15b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35905 } } peers { permanent_uuid: "c687831b599a402ea69e477d5cda0d86" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34435 } } peers { permanent_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38653 } }
I20260926 22:42:24.865710 24936 leader_election.cc:288] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d406962f79334ed0bbde8c3facb3c15b (127.11.107.194:35905), c687831b599a402ea69e477d5cda0d86 (127.11.107.193:34435)
I20260926 22:42:24.870285 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:24.866653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:58791 (local address 127.11.107.194:35905)
0926 22:42:24.866704 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:24.866706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:24.866716 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:42:24.866746 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:24.866746 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:24.866756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:24.866812 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:24.867074 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.867820 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.869281 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.869899 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.869946 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:24.869955 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:24.870171 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 22:42:24.870212 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 22:42:24.870541 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:24.865974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35905 (local address 127.11.107.195:58791)
0926 22:42:24.866341 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:24.866355 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:24.866410 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:24.866911 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:24.866913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:24.866915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:24.866991 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:24.866992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.867931 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.867932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:24.869176 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:24.869178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.870026 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.870027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:24.870089 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:24.870093 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:42:24.870407 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 22:42:24.870450 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":116,"threads_started":1}
I20260926 22:42:24.870867 24716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058e68b127ef4b35af18409f0b1ed973" candidate_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d406962f79334ed0bbde8c3facb3c15b" is_pre_election: true
I20260926 22:42:24.871040 24716 raft_consensus.cc:2454] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae5ac1901e7a4d5c832927a8f5a6b92a in term 0.
I20260926 22:42:24.871272 24785 leader_election.cc:302] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae5ac1901e7a4d5c832927a8f5a6b92a, d406962f79334ed0bbde8c3facb3c15b; no voters:
I20260926 22:42:24.871414 24936 raft_consensus.cc:2781] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:42:24.871474 24936 raft_consensus.cc:482] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:42:24.871501 24936 raft_consensus.cc:3037] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:24.872423 24936 raft_consensus.cc:504] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406962f79334ed0bbde8c3facb3c15b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35905 } } peers { permanent_uuid: "c687831b599a402ea69e477d5cda0d86" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34435 } } peers { permanent_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38653 } }
I20260926 22:42:24.872545 24936 leader_election.cc:288] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [CANDIDATE]: Term 1 election: Requested vote from peers d406962f79334ed0bbde8c3facb3c15b (127.11.107.194:35905), c687831b599a402ea69e477d5cda0d86 (127.11.107.193:34435)
I20260926 22:42:24.872810 24716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058e68b127ef4b35af18409f0b1ed973" candidate_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d406962f79334ed0bbde8c3facb3c15b"
I20260926 22:42:24.872881 24716 raft_consensus.cc:3037] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:24.873687 24716 raft_consensus.cc:2454] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae5ac1901e7a4d5c832927a8f5a6b92a in term 1.
I20260926 22:42:24.873915 24785 leader_election.cc:302] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae5ac1901e7a4d5c832927a8f5a6b92a, d406962f79334ed0bbde8c3facb3c15b; no voters:
I20260926 22:42:24.874025 24936 raft_consensus.cc:2781] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:42:24.874150 24936 raft_consensus.cc:686] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [term 1 LEADER]: Becoming Leader. State: Replica: ae5ac1901e7a4d5c832927a8f5a6b92a, State: Running, Role: LEADER
I20260926 22:42:24.874243 24936 consensus_queue.cc:237] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406962f79334ed0bbde8c3facb3c15b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35905 } } peers { permanent_uuid: "c687831b599a402ea69e477d5cda0d86" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34435 } } peers { permanent_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38653 } }
I20260926 22:42:24.874727 24927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:24.866734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:38443 (local address 127.11.107.193:34435)
0926 22:42:24.866868 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:24.866870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:24.871477 (+ 4607us) server_negotiation.cc:407] Connection header received
0926 22:42:24.871582 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:24.871583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:24.871594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:24.871622 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:24.871797 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.872466 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.873650 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.874285 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.874334 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:24.874343 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:24.874587 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 22:42:24.874637 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 22:42:24.874939 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:24.866193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:34435 (local address 127.11.107.195:38443)
0926 22:42:24.871410 (+ 5217us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:24.871421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:24.871504 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:24.871696 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:24.871697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:24.871699 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:24.871723 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:24.871724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.872564 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.872564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:24.873545 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:24.873548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.874449 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.874450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:24.874503 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:24.874505 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:42:24.874823 (+ 318us) client_negotiation.cc:240] Negotiation successful
0926 22:42:24.874860 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5206,"thread_start_us":64,"threads_started":1}
I20260926 22:42:24.875111 24427 catalog_manager.cc:5495] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a reported cstate change: term changed from 0 to 1, leader changed from <none> to ae5ac1901e7a4d5c832927a8f5a6b92a (127.11.107.195). New cstate: current_term: 1 leader_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d406962f79334ed0bbde8c3facb3c15b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c687831b599a402ea69e477d5cda0d86" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38653 } health_report { overall_health: HEALTHY } } }
I20260926 22:42:24.875988 24582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058e68b127ef4b35af18409f0b1ed973" candidate_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c687831b599a402ea69e477d5cda0d86"
I20260926 22:42:24.876112 24582 raft_consensus.cc:3037] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:24.876961 24582 raft_consensus.cc:2454] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae5ac1901e7a4d5c832927a8f5a6b92a in term 1.
I20260926 22:42:24.875994 24579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058e68b127ef4b35af18409f0b1ed973" candidate_uuid: "ae5ac1901e7a4d5c832927a8f5a6b92a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c687831b599a402ea69e477d5cda0d86" is_pre_election: true
I20260926 22:42:24.877382 24579 raft_consensus.cc:2362] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae5ac1901e7a4d5c832927a8f5a6b92a in term 1. Re-sending same reply.
I20260926 22:42:24.889909 24926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:24.886731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.11.107.195:38653)
0926 22:42:24.886819 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:24.886821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:24.886829 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:42:24.886858 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:24.886859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:24.886877 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:24.886949 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:42:24.887134 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.887843 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.888907 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:24.889173 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:24.889237 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:24.889417 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:42:24.889530 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:42:24.889594 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:24.889802 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 22:42:24.889821 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 22:42:24.891306 24629 tablet.cc:2369] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:42:24.891927 24579 raft_consensus.cc:1261] T 058e68b127ef4b35af18409f0b1ed973 P c687831b599a402ea69e477d5cda0d86 [term 1 FOLLOWER]: Refusing update from remote peer ae5ac1901e7a4d5c832927a8f5a6b92a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:42:24.892252 24716 raft_consensus.cc:1261] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b [term 1 FOLLOWER]: Refusing update from remote peer ae5ac1901e7a4d5c832927a8f5a6b92a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:42:24.892629 24936 consensus_queue.cc:1035] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d406962f79334ed0bbde8c3facb3c15b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:42:24.892792 24936 consensus_queue.cc:1035] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c687831b599a402ea69e477d5cda0d86" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:42:24.898654 24945 mvcc.cc:205] Tried to move back new op lower bound from 7333734583874482176 to 7333734583805448192. Current Snapshot: MvccSnapshot[applied={T|T < 7333734583874482176}]
I20260926 22:42:24.898861 24944 mvcc.cc:205] Tried to move back new op lower bound from 7333734583874482176 to 7333734583805448192. Current Snapshot: MvccSnapshot[applied={T|T < 7333734583874482176}]
I20260926 22:42:24.902092 24427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:42:24.902192 24427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:42:24.903090 24427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59d11a314e6b4c1c891538c4a4d49553: Sending DeleteTablet for 3 replicas of tablet 058e68b127ef4b35af18409f0b1ed973
I20260926 22:42:24.903653 24691 tablet_service.cc:1514] Processing DeleteTablet for tablet 058e68b127ef4b35af18409f0b1ed973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:24 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:59136
I20260926 22:42:24.903890 24562 tablet_service.cc:1514] Processing DeleteTablet for tablet 058e68b127ef4b35af18409f0b1ed973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:24 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:40524
I20260926 22:42:24.904129 24830 tablet_service.cc:1514] Processing DeleteTablet for tablet 058e68b127ef4b35af18409f0b1ed973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:24 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:46164
I20260926 22:42:24.904768 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 24469
I20260926 22:42:24.908169 24947 mvcc.cc:205] Tried to move back new op lower bound from 7333734583874482176 to 7333734583805448192. Current Snapshot: MvccSnapshot[applied={T|T < 7333734583874482176}]
I20260926 22:42:24.908339 24954 tablet_replica.cc:330] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a: stopping tablet replica
I20260926 22:42:24.908432 24954 raft_consensus.cc:2229] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:42:24.908519 24954 raft_consensus.cc:2258] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:42:24.908654 24952 tablet_replica.cc:330] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b: stopping tablet replica
I20260926 22:42:24.908749 24952 raft_consensus.cc:2229] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:42:24.908811 24952 raft_consensus.cc:2258] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 22:42:24.912336 24416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:34435 (error 108)
I20260926 22:42:24.912823 24954 ts_tablet_manager.cc:1905] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:42:24.917753 24952 ts_tablet_manager.cc:1905] T 058e68b127ef4b35af18409f0b1ed973 P d406962f79334ed0bbde8c3facb3c15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:42:24.917910 24954 ts_tablet_manager.cc:1918] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:42:24.917982 24954 log.cc:1196] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/058e68b127ef4b35af18409f0b1ed973
I20260926 22:42:24.918283 24954 ts_tablet_manager.cc:1939] T 058e68b127ef4b35af18409f0b1ed973 P ae5ac1901e7a4d5c832927a8f5a6b92a: Deleting consensus metadata
I20260926 22:42:24.918694 24416 catalog_manager.cc:4841] TS ae5ac1901e7a4d5c832927a8f5a6b92a (127.11.107.195:38653): tablet 058e68b127ef4b35af18409f0b1ed973 (table test-table [id=bc002145233446c0acff4521ee57d3c2]) successfully deleted
W20260926 22:42:24.919332 24925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:42:24.912940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:34435 (local address 127.0.0.1:40526)
0926 22:42:24.913015 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:24.913027 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:24.913065 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:24.919285 (+ 6220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:34435: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 22:42:24.919457 24416 catalog_manager.cc:4584] TS c687831b599a402ea69e477d5cda0d86 (127.11.107.193:34435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 058e68b127ef4b35af18409f0b1ed973: Network error: Client connection negotiation failed: client connection to 127.11.107.193:34435: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 22:42:24.920532 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 24630
W20260926 22:42:24.927572 24925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:42:24.927255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35905 (local address 127.0.0.1:59144)
0926 22:42:24.927301 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:24.927313 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:24.927351 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:24.927532 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.194:35905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 22:42:24.927990 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 24765
I20260926 22:42:24.936039 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 24397
I20260926 22:42:24.943668 11695 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 22:42:25.288371 11695 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 22:42:25.288489 11695 mini_ranger.cc:66] Stopping Ranger...
I20260926 22:42:25.632722 11695 mini_ranger.cc:68] Stopped Ranger
2026-09-26 22:42:25.632 UTC [23464] LOG: received smart shutdown request
2026-09-26 22:42:25.637 UTC [23464] LOG: background worker "logical replication launcher" (PID 23473) exited with exit code 1
2026-09-26 22:42:25.637 UTC [23468] LOG: shutting down
2026-09-26 22:42:25.790 UTC [23464] LOG: database system is shut down
2026-09-26T22:42:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46595 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:42:25 dist-test-slave-j8sq krb5kdc[24966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:42:25 dist-test-slave-j8sq krb5kdc[24966](info): set up 2 sockets
Sep 26 22:42:25 dist-test-slave-j8sq krb5kdc[24966](info): commencing operation
krb5kdc: starting...
W20260926 22:42:26.978698 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:42:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:42:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:42:27.096197 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:37271
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:37271,127.11.107.253:34553,127.11.107.252:42821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:27.178146 24979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:27.178362 24979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:27.178399 24979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:27.179515 24979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:42:27.179580 24979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:27.179606 24979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:27.179639 24979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:42:27.179663 24979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:42:27.181098 24979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:37271,127.11.107.253:34553,127.11.107.252:42821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:27.181375 24979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:27.181588 24979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:27.186319 24984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:27.186306 24985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:27.186342 24979 server_base.cc:1034] running on GCE node
W20260926 22:42:27.186306 24987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:27.186774 24979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:27.187055 24979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:27.188199 24979 hybrid_clock.cc:648] HybridClock initialized: now 1790462547188179 us; error 35 us; skew 500 ppm
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:27.193645 24979 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:42:27.194072 24979 webserver.cc:460] Webserver started at http://127.11.107.254:44933/ using document root <none> and password file <none>
I20260926 22:42:27.194273 24979 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:27.194314 24979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:27.194402 24979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:42:27.195386 24979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "c9fe84790c8b4ec19c22bbf01542b1d0"
format_stamp: "Formatted at 2026-09-26 22:42:27 on dist-test-slave-j8sq"
server_key: "ca6510afac126389c846f79960460afe"
server_key_iv: "13af8a7a1c41482ee0e8f8c8df728e50"
server_key_version: "encryptionkey@0"
I20260926 22:42:27.195756 24979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "c9fe84790c8b4ec19c22bbf01542b1d0"
format_stamp: "Formatted at 2026-09-26 22:42:27 on dist-test-slave-j8sq"
server_key: "ca6510afac126389c846f79960460afe"
server_key_iv: "13af8a7a1c41482ee0e8f8c8df728e50"
server_key_version: "encryptionkey@0"
I20260926 22:42:27.196956 24979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:42:27.197628 24994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:27.197829 24979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:42:27.197930 24979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "c9fe84790c8b4ec19c22bbf01542b1d0"
format_stamp: "Formatted at 2026-09-26 22:42:27 on dist-test-slave-j8sq"
server_key: "ca6510afac126389c846f79960460afe"
server_key_iv: "13af8a7a1c41482ee0e8f8c8df728e50"
server_key_version: "encryptionkey@0"
I20260926 22:42:27.197984 24979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:27.208626 24979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:27.210419 24979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:27.210525 24979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:27.215278 24979 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:37271
I20260926 22:42:27.215317 25046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:37271 every 8 connection(s)
I20260926 22:42:27.215670 24979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:42:27.216482 25047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:27.218034 25047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:42:27.220338 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 24979
I20260926 22:42:27.220454 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:42:27.220639 11695 external_mini_cluster.cc:1428] Setting key e04f3a85863849a3e26cddb34a6c20d4
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.107.253@KRBTEST.COM; defaulting to no policy
I20260926 22:42:27.228324 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.218809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.11.107.254:37271)
0926 22:42:27.218927 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.218931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.218960 (+ 29us) server_negotiation.cc:407] Connection header received
0926 22:42:27.219503 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.219505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.219521 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.219613 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.220144 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.221433 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.222365 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.222882 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.222992 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.225225 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.225231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.225231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.225237 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.226791 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.227184 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.227186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.227187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.227270 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.227485 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.227486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.227486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.227684 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.227771 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.227953 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.228084 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":92,"thread_start_us":52,"threads_started":1}
I20260926 22:42:27.228323 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.218649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:37271 (local address 127.0.0.1:55176)
0926 22:42:27.218812 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.218855 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:27.219343 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:27.219620 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:27.219623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:27.219901 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:27.220043 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.220048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.221524 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.221525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.222282 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.222284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.222976 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.222977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.223070 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.223531 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:27.223541 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:27.225125 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:27.226890 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.226896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.226898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.227085 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.227334 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.227335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.227335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.227419 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.227753 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:27.227758 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:27.227815 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 22:42:27.227922 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 22:42:27.228142 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":86,"thread_start_us":80,"threads_started":1}
I20260926 22:42:27.229024 25047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:42:27.229367 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.229251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:34553 (local address 127.0.0.1:59146)
0926 22:42:27.229296 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.229338 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 22:42:27.229482 24997 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.107.253:34553: connect: Connection refused (error 111)
I20260926 22:42:27.229897 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.229803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:34553 (local address 127.0.0.1:59158)
0926 22:42:27.229834 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.229875 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 22:42:27.230005 25047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.253:34553: Network error: Client connection negotiation failed: client connection to 127.11.107.253:34553: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.253.keytab.
Entry for principal kudu/127.11.107.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.253.keytab.
I20260926 22:42:27.244036 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.253:34553
--webserver_interface=127.11.107.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.253.keytab
--principal=kudu/127.11.107.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:37271,127.11.107.253:34553,127.11.107.252:42821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:42:27.279208 25047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } attempt: 1
I20260926 22:42:27.279667 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.279535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:34553 (local address 127.0.0.1:59170)
0926 22:42:27.279600 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.279639 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 22:42:27.280028 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.279934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:34553 (local address 127.0.0.1:59180)
0926 22:42:27.279979 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.280003 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 22:42:27.280112 25047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.253:34553: Network error: Client connection negotiation failed: client connection to 127.11.107.253:34553: connect: Connection refused (error 111)
W20260926 22:42:27.322273 25054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:27.322459 25054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:27.322480 25054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:27.323596 25054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:42:27.323639 25054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:27.323653 25054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:27.323665 25054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:42:27.323719 25054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:42:27.325140 25054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:37271,127.11.107.253:34553,127.11.107.252:42821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.253.keytab
--principal=kudu/127.11.107.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.253:34553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.107.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:27.325388 25054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:27.325627 25054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:27.330286 25060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:27.330339 25054 server_base.cc:1034] running on GCE node
W20260926 22:42:27.330276 25062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:27.330286 25059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:27.330760 25054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:27.331012 25054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:27.332144 25054 hybrid_clock.cc:648] HybridClock initialized: now 1790462547332133 us; error 28 us; skew 500 ppm
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:27.334812 25054 init.cc:377] Logged in from keytab as kudu/127.11.107.253@KRBTEST.COM (short username kudu)
I20260926 22:42:27.335242 25054 webserver.cc:460] Webserver started at http://127.11.107.253:35929/ using document root <none> and password file <none>
I20260926 22:42:27.335457 25054 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:27.335525 25054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:27.335678 25054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:42:27.336690 25054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/data/instance:
uuid: "9c9ec05000a448fea5f2fd907b106996"
format_stamp: "Formatted at 2026-09-26 22:42:27 on dist-test-slave-j8sq"
server_key: "7532600794285f865a15eb114c65cf07"
server_key_iv: "aa58632c9c5ce2ff81278440fb3bbe1b"
server_key_version: "encryptionkey@0"
I20260926 22:42:27.337055 25054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/wal/instance:
uuid: "9c9ec05000a448fea5f2fd907b106996"
format_stamp: "Formatted at 2026-09-26 22:42:27 on dist-test-slave-j8sq"
server_key: "7532600794285f865a15eb114c65cf07"
server_key_iv: "aa58632c9c5ce2ff81278440fb3bbe1b"
server_key_version: "encryptionkey@0"
I20260926 22:42:27.338479 25054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:42:27.339175 25069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:27.339352 25054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:42:27.339455 25054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/wal
uuid: "9c9ec05000a448fea5f2fd907b106996"
format_stamp: "Formatted at 2026-09-26 22:42:27 on dist-test-slave-j8sq"
server_key: "7532600794285f865a15eb114c65cf07"
server_key_iv: "aa58632c9c5ce2ff81278440fb3bbe1b"
server_key_version: "encryptionkey@0"
I20260926 22:42:27.339527 25054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:27.348241 25047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } attempt: 2
I20260926 22:42:27.348658 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.348540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:34553 (local address 127.0.0.1:59196)
0926 22:42:27.348594 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.348629 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 22:42:27.348925 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.348867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:34553 (local address 127.0.0.1:59212)
0926 22:42:27.348890 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.348910 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 22:42:27.349030 25047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.253:34553: Network error: Client connection negotiation failed: client connection to 127.11.107.253:34553: connect: Connection refused (error 111)
I20260926 22:42:27.353904 25054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:27.355566 25054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:27.355722 25054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:27.359680 25054 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.253:34553
I20260926 22:42:27.359748 25121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.253:34553 every 8 connection(s)
I20260926 22:42:27.360059 25054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/data/info.pb
I20260926 22:42:27.360872 25122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:27.362318 25122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.253@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:27.368297 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 25054
I20260926 22:42:27.368409 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-1/wal/instance
I20260926 22:42:27.368579 11695 external_mini_cluster.cc:1428] Setting key 5f184a2dbe0275ac703fc13b664fe52d
I20260926 22:42:27.370771 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.362911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.11.107.254:37271)
0926 22:42:27.362972 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.362974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.363172 (+ 198us) server_negotiation.cc:407] Connection header received
0926 22:42:27.363564 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.363564 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.363576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.363635 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.364155 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.364612 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.365555 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.365754 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.365808 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.367836 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.367837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.367838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.367844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.369242 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.369741 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.369742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.369743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.369831 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.370090 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.370090 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.370091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.370235 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.370307 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.370563 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.370648 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:42:27.370792 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.362902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:37271 (local address 127.0.0.1:55186)
0926 22:42:27.363067 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.363098 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:27.363492 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:27.363680 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:27.363682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:27.363915 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:27.364087 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.364090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.364744 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.364744 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.365479 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.365480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.365828 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.365829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.365989 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.366468 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:27.366476 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:27.367762 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:27.369323 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.369329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.369331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.369652 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.369922 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.369927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.369928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.370023 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.370304 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:27.370309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:27.370423 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 22:42:27.370519 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 22:42:27.370650 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
I20260926 22:42:27.371327 25122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.253@KRBTEST.COM for kudu/127.11.107.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.107.252@KRBTEST.COM; defaulting to no policy
I20260926 22:42:27.378443 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.371558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:34553 (local address 127.0.0.1:59224)
0926 22:42:27.371603 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.371613 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:27.371649 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:27.371908 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:27.371909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:27.372068 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:27.372125 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.372127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.372934 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.372936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.373816 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.373819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.374276 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.374277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.374355 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.374679 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:27.374684 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:27.375868 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:27.377543 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.377544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.377544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.377737 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.377966 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.377967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.377967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.378017 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.378240 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:27.378240 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:27.378271 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 22:42:27.378318 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:42:27.378386 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 22:42:27.378543 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.371647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.11.107.253:34553)
0926 22:42:27.371768 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.371772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.371788 (+ 16us) server_negotiation.cc:407] Connection header received
0926 22:42:27.371805 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.371807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.371818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.371879 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.372201 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.372846 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.373929 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.374198 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.374253 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.375955 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.375962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.375963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.375970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.377469 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.377803 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.377806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.377806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.377909 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.378063 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.378063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.378064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.378180 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.378238 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.378340 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.378433 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":96,"thread_start_us":75,"threads_started":1}
I20260926 22:42:27.379103 25122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.252@KRBTEST.COM" created.
I20260926 22:42:27.379460 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.379331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:42821 (local address 127.0.0.1:57114)
0926 22:42:27.379376 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.379438 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 22:42:27.379585 25073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
I20260926 22:42:27.380033 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.379946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:42821 (local address 127.0.0.1:57130)
0926 22:42:27.379990 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.380016 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 22:42:27.380165 25122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:42821: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.252.keytab.
Entry for principal kudu/127.11.107.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.252.keytab.
I20260926 22:42:27.392437 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.252:42821
--webserver_interface=127.11.107.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.252.keytab
--principal=kudu/127.11.107.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:37271,127.11.107.253:34553,127.11.107.252:42821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:42:27.429388 25122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } attempt: 1
I20260926 22:42:27.429904 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.429720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:42821 (local address 127.0.0.1:57140)
0926 22:42:27.429781 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.429821 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 22:42:27.430220 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.430126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:42821 (local address 127.0.0.1:57144)
0926 22:42:27.430170 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.430197 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 22:42:27.430315 25122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:42821: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
I20260926 22:42:27.440218 25047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } attempt: 3
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for kudu/127.11.107.253@KRBTEST.COM
I20260926 22:42:27.447822 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.440594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:34553 (local address 127.0.0.1:59232)
0926 22:42:27.440742 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.440754 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:27.440818 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:27.440991 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:27.440993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:27.441136 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:27.441216 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.441219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.442226 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.442228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.442794 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.442796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.443293 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.443294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.443356 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.443847 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:27.443852 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:27.445199 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:27.446948 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.446949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.446949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.447106 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.447317 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.447318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.447318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.447365 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.447605 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:27.447606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:27.447641 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 22:42:27.447692 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:42:27.447758 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260926 22:42:27.447876 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.440698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.11.107.253:34553)
0926 22:42:27.440756 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.440758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.440820 (+ 62us) server_negotiation.cc:407] Connection header received
0926 22:42:27.440881 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.440881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.440894 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.440968 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.441307 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.442145 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.442886 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.443217 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.443293 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.445286 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.445288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.445288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.445296 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.446872 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.447176 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.447177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.447178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.447249 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.447421 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.447422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.447422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.447547 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.447610 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.447714 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.447788 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 22:42:27.448246 25047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:42:27.448582 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.448478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:42821 (local address 127.0.0.1:57146)
0926 22:42:27.448522 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.448557 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 22:42:27.448979 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.448905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:42821 (local address 127.0.0.1:57158)
0926 22:42:27.448943 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.448964 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 22:42:27.449084 25047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:42821: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
W20260926 22:42:27.473951 25129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:27.474128 25129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:27.474148 25129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:27.475188 25129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:42:27.475220 25129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:27.475234 25129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:27.475246 25129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:42:27.475258 25129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:42:27.476599 25129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:37271,127.11.107.253:34553,127.11.107.252:42821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.252.keytab
--principal=kudu/127.11.107.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.252:42821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.107.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:27.476773 25129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:27.476922 25129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:42:27.480221 25047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } attempt: 1
I20260926 22:42:27.480579 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.480468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:42821 (local address 127.0.0.1:57172)
0926 22:42:27.480510 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.480547 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 22:42:27.480938 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.480855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:42821 (local address 127.0.0.1:57188)
0926 22:42:27.480889 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.480921 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 22:42:27.481035 25047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:42821: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
W20260926 22:42:27.481308 25134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:27.481310 25135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:27.481305 25137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:27.482120 25129 server_base.cc:1034] running on GCE node
I20260926 22:42:27.482369 25129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:27.482669 25129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:27.483819 25129 hybrid_clock.cc:648] HybridClock initialized: now 1790462547483808 us; error 24 us; skew 500 ppm
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:27.486143 25129 init.cc:377] Logged in from keytab as kudu/127.11.107.252@KRBTEST.COM (short username kudu)
I20260926 22:42:27.486579 25129 webserver.cc:460] Webserver started at http://127.11.107.252:41579/ using document root <none> and password file <none>
I20260926 22:42:27.486800 25129 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:27.486845 25129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:27.486955 25129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:42:27.487955 25129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/data/instance:
uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1"
format_stamp: "Formatted at 2026-09-26 22:42:27 on dist-test-slave-j8sq"
server_key: "ad72213f5510812c298f8169d60b0676"
server_key_iv: "2f2a0a0c2f5328de667bea963eb451a9"
server_key_version: "encryptionkey@0"
I20260926 22:42:27.488299 25129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/wal/instance:
uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1"
format_stamp: "Formatted at 2026-09-26 22:42:27 on dist-test-slave-j8sq"
server_key: "ad72213f5510812c298f8169d60b0676"
server_key_iv: "2f2a0a0c2f5328de667bea963eb451a9"
server_key_version: "encryptionkey@0"
I20260926 22:42:27.489543 25129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 22:42:27.490268 25144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:27.490437 25129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:27.490530 25129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/wal
uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1"
format_stamp: "Formatted at 2026-09-26 22:42:27 on dist-test-slave-j8sq"
server_key: "ad72213f5510812c298f8169d60b0676"
server_key_iv: "2f2a0a0c2f5328de667bea963eb451a9"
server_key_version: "encryptionkey@0"
I20260926 22:42:27.490608 25129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:27.498144 25129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:27.498458 25122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } attempt: 2
I20260926 22:42:27.498802 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.498687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:42821 (local address 127.0.0.1:57192)
0926 22:42:27.498742 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.498777 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 22:42:27.499136 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.499044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:42821 (local address 127.0.0.1:57204)
0926 22:42:27.499091 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.499114 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 22:42:27.499274 25122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:42821: Network error: Client connection negotiation failed: client connection to 127.11.107.252:42821: connect: Connection refused (error 111)
I20260926 22:42:27.500376 25129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:27.500531 25129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:27.504186 25129 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.252:42821
I20260926 22:42:27.504246 25196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.252:42821 every 8 connection(s)
I20260926 22:42:27.504583 25129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/data/info.pb
I20260926 22:42:27.505443 25197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:27.506592 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 25129
I20260926 22:42:27.506697 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462400949426-11695-0/minicluster-data/master-2/wal/instance
I20260926 22:42:27.506894 11695 external_mini_cluster.cc:1428] Setting key 87580b157f3aab0603a5ab43fc212c5c
I20260926 22:42:27.507143 25197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.252@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.252@KRBTEST.COM
I20260926 22:42:27.516039 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.507847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.11.107.254:37271)
0926 22:42:27.507911 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.507913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.508038 (+ 125us) server_negotiation.cc:407] Connection header received
0926 22:42:27.508220 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.508221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.508233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.508280 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.508792 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.509260 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.510217 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.510409 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.510513 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.512768 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.512770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.512770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.512777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.514453 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.514928 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.514929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.514930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.514990 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.515316 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.515317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.515317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.515463 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.515522 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.515850 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.515938 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 22:42:27.516234 25200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.507364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.11.107.252:42821)
0926 22:42:27.507552 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.507560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.507592 (+ 32us) server_negotiation.cc:407] Connection header received
0926 22:42:27.508123 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.508129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.508149 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.508262 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.508845 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.509591 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.510403 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.510718 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.510799 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.513275 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.513282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.513282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.513289 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.514950 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.515328 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.515331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.515331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.515379 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.515584 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.515585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.515585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.515719 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.515855 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.515929 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.516032 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260926 22:42:27.516326 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.507791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:37271 (local address 127.0.0.1:55196)
0926 22:42:27.507924 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.507962 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:27.508129 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:27.508307 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:27.508310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:27.508561 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:27.508704 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.508708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.509349 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.509350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.510118 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.510120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.510518 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.510520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.510644 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.511095 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:27.511105 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:27.512680 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:27.514542 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.514547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.514549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.514855 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.515077 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.515078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.515078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.515224 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.515536 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:27.515543 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:27.515687 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 22:42:27.515801 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 22:42:27.515972 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260926 22:42:27.516922 25197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.252@KRBTEST.COM for kudu/127.11.107.253@KRBTEST.COM
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:27.523690 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.517154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:34553 (local address 127.0.0.1:59248)
0926 22:42:27.517208 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.517219 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:27.517258 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:27.517373 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:27.517373 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:27.517524 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:27.517571 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.517573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.518257 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.518258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.518733 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.518735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.519095 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.519096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.519161 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.519555 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:27.519560 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:27.520751 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:27.522667 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.522668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.522668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.522823 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.523035 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.523036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.523036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.523083 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.523374 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:27.523375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:27.523419 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 22:42:27.523501 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 22:42:27.523590 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 22:42:27.523725 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.517225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.11.107.253:34553)
0926 22:42:27.517265 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.517267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.517286 (+ 19us) server_negotiation.cc:407] Connection header received
0926 22:42:27.517311 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.517311 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.517325 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.517358 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.517642 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.518163 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.518817 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.519009 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.519096 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.520852 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.520854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.520854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.520860 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.522578 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.522896 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.522897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.522898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.522956 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.523145 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.523146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.523147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.523293 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.523355 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.523548 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.523631 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:42:27.524189 25197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:42:27.524513 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.517025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.11.107.254:37271)
0926 22:42:27.517224 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.517226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.517233 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:42:27.517253 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.517254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.517263 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.517315 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.517895 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.518325 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.518986 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.519167 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.519240 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.521490 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.521492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.521492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.521499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.523182 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.523524 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.523525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.523525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.523588 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.523812 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.523813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.523814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.523937 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.524103 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.524312 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.524389 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.252@KRBTEST.COM for kudu/127.11.107.252@KRBTEST.COM
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.253@KRBTEST.COM
I20260926 22:42:27.531028 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.524465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:42821 (local address 127.0.0.1:57218)
0926 22:42:27.524496 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.524507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:27.524543 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:27.524716 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:27.524717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:27.524877 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:27.524908 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.524910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.525806 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.525806 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.526328 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.526330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.526768 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.526770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.526825 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.527152 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:27.527156 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:27.528379 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:27.530086 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.530087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.530087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.530244 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.530483 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.530483 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.530484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.530550 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.530819 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:27.530820 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:27.530851 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 22:42:27.530902 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:42:27.530972 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 22:42:27.531147 25200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.524548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57218 (local address 127.11.107.252:42821)
0926 22:42:27.524583 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.524585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.524593 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:42:27.524617 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.524618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.524629 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.524685 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.524987 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.525725 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.526423 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.526685 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.526757 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.528459 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.528460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.528460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.528467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.530002 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.530345 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.530346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.530346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.530391 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.530616 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.530617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.530618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.530750 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.530819 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.530931 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.531037 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":17}
I20260926 22:42:27.532368 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.525242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.11.107.253:34553)
0926 22:42:27.525275 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.525277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.525304 (+ 27us) server_negotiation.cc:407] Connection header received
0926 22:42:27.525321 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.525322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.525349 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.525385 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.525952 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.526387 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.527199 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.527375 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.527427 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.529589 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.529590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.529591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.529597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.531196 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.531603 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.531604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.531604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.531655 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.531862 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.531863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.531863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.531978 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.532154 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.532189 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.532260 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:42:27.532862 25197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1: Bootstrap starting.
I20260926 22:42:27.533882 25197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:27.534003 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:42:27.534176 25197 log.cc:824] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:27.534919 25197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1: No bootstrap required, opened a new log
I20260926 22:42:27.537294 25197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } }
I20260926 22:42:27.537462 25197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:27.537498 25197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9e7caf84cb4b34bd0e50ace2066ad1, State: Initialized, Role: FOLLOWER
I20260926 22:42:27.537642 25197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } }
I20260926 22:42:27.537964 25197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:42:27.538200 25210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } } }
I20260926 22:42:27.538312 25210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 22:42:27.540401 25221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:42:27.540503 25221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:42:27.540133 25129 master_runner.cc:186] Error getting master registration for 127.11.107.254:37271: OK, instance_id { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" instance_seqno: 1790462547213979 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 22:42:27.543262 25129 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 22:42:27.543915 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.535357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59262 (local address 127.11.107.253:34553)
0926 22:42:27.535392 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.535395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.535830 (+ 435us) server_negotiation.cc:407] Connection header received
0926 22:42:27.535930 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.535932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.535941 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.535995 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.536451 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.536896 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.537655 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.537976 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.538070 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.539641 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.539642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.539643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.539649 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.542596 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.543049 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.543050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.543051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.543109 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.543303 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.543304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.543304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.543429 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.543537 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.543732 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.543799 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:42:27.546370 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.535701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.11.107.254:37271)
0926 22:42:27.535742 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.535744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.535754 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:42:27.535805 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.535806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.535816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.535872 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.536550 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.537238 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.538254 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.538866 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.538911 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.540666 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.540667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.540667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.540674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.544049 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.544712 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.544714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.544714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.544770 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.545666 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.545667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.545667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.545803 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.545913 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.546209 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.546287 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:42:27.547577 25200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.535541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.11.107.252:42821)
0926 22:42:27.535575 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.535577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.539533 (+ 3956us) server_negotiation.cc:407] Connection header received
0926 22:42:27.539635 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.539636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.539646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.539675 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.540296 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.541002 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.542095 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.542389 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.542431 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.543808 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.543810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.543811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.543818 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.545437 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.546778 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.546779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.546779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.546841 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.547040 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.547041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.547041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.547168 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.547264 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.547402 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.547478 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:42:27.556170 25047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } attempt: 2
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for kudu/127.11.107.252@KRBTEST.COM
I20260926 22:42:27.563252 25200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.556522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.11.107.252:42821)
0926 22:42:27.556576 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.556577 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.556609 (+ 32us) server_negotiation.cc:407] Connection header received
0926 22:42:27.556625 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.556625 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.556633 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.556670 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.556976 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.557418 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.558406 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.558582 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.558637 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.560594 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.560595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.560596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.560604 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.562114 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.562487 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.562489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.562489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.562549 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.562745 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.562746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.562746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.562851 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.562931 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.563095 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.563162 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 22:42:27.563254 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.556458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:42821 (local address 127.0.0.1:57244)
0926 22:42:27.556515 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.556527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:27.556571 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:27.556688 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:27.556689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:27.556863 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:27.556908 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.556910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.557491 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.557492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.558324 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.558326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.558654 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.558655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.558721 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.559128 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:27.559134 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:27.560433 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:27.562228 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.562229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.562230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.562401 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.562616 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.562616 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.562617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.562682 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.562919 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:27.562919 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:27.562964 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 22:42:27.563068 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 22:42:27.563162 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 22:42:27.565032 25047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0: Bootstrap starting.
I20260926 22:42:27.565786 25047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:27.566128 25047 log.cc:824] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:27.567028 25047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0: No bootstrap required, opened a new log
I20260926 22:42:27.569041 25047 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } }
I20260926 22:42:27.569186 25047 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:27.569221 25047 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fe84790c8b4ec19c22bbf01542b1d0, State: Initialized, Role: FOLLOWER
I20260926 22:42:27.569749 25047 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } }
I20260926 22:42:27.570127 25047 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:42:27.570104 25224 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } } }
I20260926 22:42:27.570218 25224 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 22:42:27.572042 25235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:42:27.572144 25235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:42:27.572769 24979 master_runner.cc:186] Error getting master registration for 127.11.107.253:34553: OK, instance_id { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" instance_seqno: 1790462547358608 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 22:42:27.574836 24979 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 22:42:27.590420 25122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } attempt: 3
Sep 26 22:42:27 dist-test-slave-j8sq krb5kdc[24966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462547, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.253@KRBTEST.COM for kudu/127.11.107.252@KRBTEST.COM
I20260926 22:42:27.597426 25200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.590755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.11.107.252:42821)
0926 22:42:27.590810 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:27.590812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:27.590820 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:42:27.590893 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:27.590893 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:27.590903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:27.590974 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:27.591260 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.591729 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.592407 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.592592 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.592677 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.594749 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:27.594750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:27.594751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:27.594758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:27.596284 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.596649 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.596650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.596651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.596721 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:27.596922 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:27.596923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:27.596924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:27.597046 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:27.597119 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:27.597278 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 22:42:27.597341 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:42:27.597426 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:27.590725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:42821 (local address 127.0.0.1:57258)
0926 22:42:27.590777 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:27.590788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:27.590831 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:27.590978 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:27.590979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:27.591139 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:27.591185 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.591186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.591816 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.591817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.592321 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:27.592323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:27.592684 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:27.592686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:27.592746 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:27.593274 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:27.593279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:27.594661 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:27.596361 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.596362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.596363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.596571 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.596792 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:27.596792 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:27.596793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:27.596853 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:27.597123 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:27.597124 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:27.597171 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 22:42:27.597242 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 22:42:27.597333 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 22:42:27.598869 25122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996: Bootstrap starting.
I20260926 22:42:27.599504 25122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:27.599951 25122 log.cc:824] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:27.600728 25122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996: No bootstrap required, opened a new log
I20260926 22:42:27.602286 25122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } }
I20260926 22:42:27.602412 25122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:27.602455 25122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c9ec05000a448fea5f2fd907b106996, State: Initialized, Role: FOLLOWER
I20260926 22:42:27.602571 25122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } }
I20260926 22:42:27.602828 25238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } } }
I20260926 22:42:27.602890 25122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:42:27.602953 25238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 22:42:27.604696 25249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:42:27.604799 25249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 22:42:27.606297 25075 tablet.cc:2369] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:42:27.606904 25054 master_runner.cc:263] No leader master found from master 9c9ec05000a448fea5f2fd907b106996
I20260926 22:42:27.606956 25054 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 22:42:27.709367 25238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:42:27.709496 25238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } }
I20260926 22:42:27.709771 25238 leader_election.cc:288] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9fe84790c8b4ec19c22bbf01542b1d0 (127.11.107.254:37271), 5c9e7caf84cb4b34bd0e50ace2066ad1 (127.11.107.252:42821)
I20260926 22:42:27.710031 25021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c9ec05000a448fea5f2fd907b106996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" is_pre_election: true
I20260926 22:42:27.710186 25021 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c9ec05000a448fea5f2fd907b106996 in term 0.
I20260926 22:42:27.710695 25171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c9ec05000a448fea5f2fd907b106996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" is_pre_election: true
I20260926 22:42:27.710839 25171 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c9ec05000a448fea5f2fd907b106996 in term 0.
I20260926 22:42:27.711113 25073 leader_election.cc:302] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9e7caf84cb4b34bd0e50ace2066ad1, 9c9ec05000a448fea5f2fd907b106996; no voters:
W20260926 22:42:27.711199 25000 tablet.cc:2369] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:42:27.711253 25238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:42:27.711293 25238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:42:27.711313 25238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:27.712318 25238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } }
I20260926 22:42:27.712436 25238 leader_election.cc:288] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [CANDIDATE]: Term 1 election: Requested vote from peers c9fe84790c8b4ec19c22bbf01542b1d0 (127.11.107.254:37271), 5c9e7caf84cb4b34bd0e50ace2066ad1 (127.11.107.252:42821)
I20260926 22:42:27.712729 25171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c9ec05000a448fea5f2fd907b106996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1"
I20260926 22:42:27.712805 25171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:27.713723 25171 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c9ec05000a448fea5f2fd907b106996 in term 1.
I20260926 22:42:27.713964 25073 leader_election.cc:302] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9e7caf84cb4b34bd0e50ace2066ad1, 9c9ec05000a448fea5f2fd907b106996; no voters:
I20260926 22:42:27.714282 25021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c9ec05000a448fea5f2fd907b106996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0"
I20260926 22:42:27.714360 25021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:27.715209 25021 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c9ec05000a448fea5f2fd907b106996 in term 1.
I20260926 22:42:27.716385 25238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:42:27.716531 25238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [term 1 LEADER]: Becoming Leader. State: Replica: 9c9ec05000a448fea5f2fd907b106996, State: Running, Role: LEADER
I20260926 22:42:27.716614 25238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } }
I20260926 22:42:27.716964 25238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c9ec05000a448fea5f2fd907b106996. Latest consensus state: current_term: 1 leader_uuid: "9c9ec05000a448fea5f2fd907b106996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } } }
I20260926 22:42:27.717031 25238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:27.717813 25253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:42:27.718047 25253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:42:27.719529 25171 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [term 1 FOLLOWER]: Refusing update from remote peer 9c9ec05000a448fea5f2fd907b106996: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:42:27.719903 25251 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:42:27.721125 25210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c9ec05000a448fea5f2fd907b106996. Latest consensus state: current_term: 1 leader_uuid: "9c9ec05000a448fea5f2fd907b106996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } } }
I20260926 22:42:27.721216 25210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:42:27.721582 25021 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [term 1 FOLLOWER]: Refusing update from remote peer 9c9ec05000a448fea5f2fd907b106996: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:42:27.721817 25238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:42:27.722863 25224 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c9ec05000a448fea5f2fd907b106996. Latest consensus state: current_term: 1 leader_uuid: "9c9ec05000a448fea5f2fd907b106996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } } }
I20260926 22:42:27.722939 25224 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:42:27.724047 25251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c9ec05000a448fea5f2fd907b106996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } } }
I20260926 22:42:27.724134 25251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:27.724584 25210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c9ec05000a448fea5f2fd907b106996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } } }
I20260926 22:42:27.724653 25210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:42:27.724952 25254 mvcc.cc:205] Tried to move back new op lower bound from 7333734595456962560 to 7333734595447676928. Current Snapshot: MvccSnapshot[applied={T|T < 7333734595456962560}]
I20260926 22:42:27.725071 25255 mvcc.cc:205] Tried to move back new op lower bound from 7333734595456962560 to 7333734595447676928. Current Snapshot: MvccSnapshot[applied={T|T < 7333734595456962560}]
I20260926 22:42:27.726022 25238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c9ec05000a448fea5f2fd907b106996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } } }
I20260926 22:42:27.726089 25238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:27.726502 25224 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c9ec05000a448fea5f2fd907b106996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9fe84790c8b4ec19c22bbf01542b1d0" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37271 } } peers { permanent_uuid: "9c9ec05000a448fea5f2fd907b106996" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 34553 } } peers { permanent_uuid: "5c9e7caf84cb4b34bd0e50ace2066ad1" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 42821 } } }
I20260926 22:42:27.726572 25224 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:42:27.726794 25257 mvcc.cc:205] Tried to move back new op lower bound from 7333734595456962560 to 7333734595447676928. Current Snapshot: MvccSnapshot[applied={T|T < 7333734595456962560}]
I20260926 22:42:27.727365 25253 catalog_manager.cc:1347] Generated new cluster ID: 32b316e5cd84414384ea3ec0ed010798
I20260926 22:42:27.727404 25253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:42:27.742708 25253 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:42:27.743433 25253 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:42:27.755210 25253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c9ec05000a448fea5f2fd907b106996: Generated new TSK 0
I20260926 22:42:27.755386 25253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:42:28.541014 25221 catalog_manager.cc:1259] Loaded cluster ID: 32b316e5cd84414384ea3ec0ed010798
I20260926 22:42:28.541096 25221 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1: loading cluster ID for follower catalog manager: success
I20260926 22:42:28.542007 25221 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1: acquiring CA information for follower catalog manager: success
I20260926 22:42:28.542322 25221 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c9e7caf84cb4b34bd0e50ace2066ad1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:42:28.572628 25235 catalog_manager.cc:1259] Loaded cluster ID: 32b316e5cd84414384ea3ec0ed010798
I20260926 22:42:28.572708 25235 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0: loading cluster ID for follower catalog manager: success
I20260926 22:42:28.573545 25235 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0: acquiring CA information for follower catalog manager: success
I20260926 22:42:28.573887 25235 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c9fe84790c8b4ec19c22bbf01542b1d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:42:28.800671 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 24979
I20260926 22:42:28.806100 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 25054
I20260926 22:42:28.810824 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 25129
2026-09-26T22:42:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2900 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:42:28 dist-test-slave-j8sq krb5kdc[25268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:42:28 dist-test-slave-j8sq krb5kdc[25268](info): set up 2 sockets
Sep 26 22:42:28 dist-test-slave-j8sq krb5kdc[25268](info): commencing operation
krb5kdc: starting...
W20260926 22:42:29.838856 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:42:29 dist-test-slave-j8sq krb5kdc[25268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:42:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:42:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:42:29.952616 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:35901
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:38991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:35901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:30.030962 25281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:30.031157 25281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:30.031185 25281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:30.032261 25281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:42:30.032311 25281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:30.032326 25281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:30.032338 25281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:42:30.032351 25281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:42:30.033717 25281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:38991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:35901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:35901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:30.033963 25281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:30.034139 25281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:30.038805 25286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:30.038808 25289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:30.038867 25287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:30.038901 25281 server_base.cc:1034] running on GCE node
I20260926 22:42:30.039476 25281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:30.039754 25281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:30.040922 25281 hybrid_clock.cc:648] HybridClock initialized: now 1790462550040909 us; error 35 us; skew 500 ppm
Sep 26 22:42:30 dist-test-slave-j8sq krb5kdc[25268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:30.043417 25281 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:42:30.043889 25281 webserver.cc:460] Webserver started at http://127.11.107.254:38099/ using document root <none> and password file <none>
I20260926 22:42:30.044111 25281 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:30.044186 25281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:30.044353 25281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:42:30.045431 25281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "40a32698e64f47d786aea7867771de51"
format_stamp: "Formatted at 2026-09-26 22:42:30 on dist-test-slave-j8sq"
server_key: "289a59e04889b2aee0d38b6fe93de189"
server_key_iv: "324956e3b182457fdf943bc7293a981f"
server_key_version: "encryptionkey@0"
I20260926 22:42:30.045789 25281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "40a32698e64f47d786aea7867771de51"
format_stamp: "Formatted at 2026-09-26 22:42:30 on dist-test-slave-j8sq"
server_key: "289a59e04889b2aee0d38b6fe93de189"
server_key_iv: "324956e3b182457fdf943bc7293a981f"
server_key_version: "encryptionkey@0"
I20260926 22:42:30.047302 25281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:30.047962 25296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:30.048111 25281 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:42:30.048215 25281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "40a32698e64f47d786aea7867771de51"
format_stamp: "Formatted at 2026-09-26 22:42:30 on dist-test-slave-j8sq"
server_key: "289a59e04889b2aee0d38b6fe93de189"
server_key_iv: "324956e3b182457fdf943bc7293a981f"
server_key_version: "encryptionkey@0"
I20260926 22:42:30.048298 25281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:30.074959 25281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:30.077416 25281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:30.077591 25281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:30.081779 25281 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:35901
I20260926 22:42:30.081818 25348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:35901 every 8 connection(s)
I20260926 22:42:30.082234 25281 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:42:30.083091 25349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:30.085383 25349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51: Bootstrap starting.
I20260926 22:42:30.086208 25349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:30.086537 25349 log.cc:824] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:30.087042 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 25281
I20260926 22:42:30.087142 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:42:30.087334 25349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51: No bootstrap required, opened a new log
I20260926 22:42:30.087347 11695 external_mini_cluster.cc:1428] Setting key 02b073ca62a39884caf9a145c317cba3
I20260926 22:42:30.089694 25349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a32698e64f47d786aea7867771de51" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35901 } }
I20260926 22:42:30.089877 25349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:30.089936 25349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a32698e64f47d786aea7867771de51, State: Initialized, Role: FOLLOWER
I20260926 22:42:30.090065 25349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a32698e64f47d786aea7867771de51" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35901 } }
I20260926 22:42:30.090145 25349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:42:30.090189 25349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:42:30.090237 25349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:30.090773 25349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a32698e64f47d786aea7867771de51" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35901 } }
I20260926 22:42:30.090886 25349 leader_election.cc:302] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40a32698e64f47d786aea7867771de51; no voters:
I20260926 22:42:30.091056 25349 leader_election.cc:288] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:42:30.091076 25354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:42:30.091178 25354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [term 1 LEADER]: Becoming Leader. State: Replica: 40a32698e64f47d786aea7867771de51, State: Running, Role: LEADER
I20260926 22:42:30.091272 25354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a32698e64f47d786aea7867771de51" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35901 } }
I20260926 22:42:30.091301 25349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:42:30.091584 25354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40a32698e64f47d786aea7867771de51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a32698e64f47d786aea7867771de51" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35901 } } }
I20260926 22:42:30.091734 25354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:30.091915 25354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40a32698e64f47d786aea7867771de51. Latest consensus state: current_term: 1 leader_uuid: "40a32698e64f47d786aea7867771de51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a32698e64f47d786aea7867771de51" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35901 } } }
I20260926 22:42:30.092051 25354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:30.092399 25362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:42:30.092952 25362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:42:30.094898 25362 catalog_manager.cc:1347] Generated new cluster ID: 7d3e5f9f40ef4cd79a82c7e8e0611fb4
I20260926 22:42:30.094955 25362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:42:30 dist-test-slave-j8sq krb5kdc[25268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:30.098754 25353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:30.087817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.11.107.254:35901)
0926 22:42:30.087981 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:30.087989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:30.088025 (+ 36us) server_negotiation.cc:407] Connection header received
0926 22:42:30.088613 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:30.088619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:30.088702 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:30.088814 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:30.089597 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:30.090908 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:30.091943 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:30.092525 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:30.092610 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:30.095581 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:30.095585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:30.095586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:30.095597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:30.097275 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:30.097651 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:30.097655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:30.097655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:30.097743 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:30.097985 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:30.097986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:30.097987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:30.098221 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:30.098392 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:30.098463 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 22:42:30.098599 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260926 22:42:30.099855 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:42:30.109654 25362 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:42:30.110216 25362 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:42:30.118098 25362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40a32698e64f47d786aea7867771de51: Generated new TSK 0
I20260926 22:42:30.118291 25362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:42:30.153496 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 25281
2026-09-26T22:42:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1341 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:42:30 dist-test-slave-j8sq krb5kdc[25376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:42:30 dist-test-slave-j8sq krb5kdc[25376](info): set up 2 sockets
Sep 26 22:42:30 dist-test-slave-j8sq krb5kdc[25376](info): commencing operation
krb5kdc: starting...
W20260926 22:42:31.182837 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:42:31 dist-test-slave-j8sq krb5kdc[25376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:42:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:42:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:42:31.302168 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:44595
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:44595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:31.380782 25389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:31.381011 25389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:31.381050 25389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:31.382231 25389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:42:31.382287 25389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:31.382310 25389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:31.382332 25389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:42:31.382357 25389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:42:31.383764 25389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:44595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:44595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:31.384011 25389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:31.384224 25389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:31.388875 25394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:31.388931 25389 server_base.cc:1034] running on GCE node
W20260926 22:42:31.388888 25395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:31.388882 25397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:31.389376 25389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:31.389619 25389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:31.390781 25389 hybrid_clock.cc:648] HybridClock initialized: now 1790462551390745 us; error 50 us; skew 500 ppm
Sep 26 22:42:31 dist-test-slave-j8sq krb5kdc[25376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:31.393163 25389 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:42:31.393581 25389 webserver.cc:460] Webserver started at http://127.11.107.254:41455/ using document root <none> and password file <none>
I20260926 22:42:31.393790 25389 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:31.393833 25389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:31.394003 25389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:42:31.394987 25389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "7c379db711dc4fe6a2f5edeadbd30d69"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "276014c64d4897982bb8e6c3f0cf5e76"
server_key_iv: "fd5f28b92dc40635457098d419980df6"
server_key_version: "encryptionkey@0"
I20260926 22:42:31.395349 25389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "7c379db711dc4fe6a2f5edeadbd30d69"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "276014c64d4897982bb8e6c3f0cf5e76"
server_key_iv: "fd5f28b92dc40635457098d419980df6"
server_key_version: "encryptionkey@0"
I20260926 22:42:31.396719 25389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:42:31.397403 25404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:31.397557 25389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:42:31.397652 25389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "7c379db711dc4fe6a2f5edeadbd30d69"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "276014c64d4897982bb8e6c3f0cf5e76"
server_key_iv: "fd5f28b92dc40635457098d419980df6"
server_key_version: "encryptionkey@0"
I20260926 22:42:31.397735 25389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:31.408622 25389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:31.410423 25389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:31.410578 25389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:31.414290 25389 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:44595
I20260926 22:42:31.414321 25456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:44595 every 8 connection(s)
I20260926 22:42:31.414649 25389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:42:31.415585 25457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:31.416115 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 25389
I20260926 22:42:31.416195 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:42:31.416399 11695 external_mini_cluster.cc:1428] Setting key 0d4a3eec6762bdb20192cce9dae5745c
I20260926 22:42:31.418488 25457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Bootstrap starting.
I20260926 22:42:31.419157 25457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:31.419473 25457 log.cc:824] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:31.420142 25457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: No bootstrap required, opened a new log
I20260926 22:42:31.422364 25457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } }
I20260926 22:42:31.422488 25457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:31.422510 25457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c379db711dc4fe6a2f5edeadbd30d69, State: Initialized, Role: FOLLOWER
I20260926 22:42:31.422649 25457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } }
I20260926 22:42:31.422766 25457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:42:31.422806 25457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:42:31.422863 25457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 22:42:31 dist-test-slave-j8sq krb5kdc[25376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:31.423527 25457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } }
I20260926 22:42:31.423642 25457 leader_election.cc:302] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c379db711dc4fe6a2f5edeadbd30d69; no voters:
I20260926 22:42:31.423804 25457 leader_election.cc:288] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:42:31.423877 25462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:42:31.424029 25462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 1 LEADER]: Becoming Leader. State: Replica: 7c379db711dc4fe6a2f5edeadbd30d69, State: Running, Role: LEADER
I20260926 22:42:31.424152 25457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:42:31.424175 25462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } }
I20260926 22:42:31.424546 25462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c379db711dc4fe6a2f5edeadbd30d69. Latest consensus state: current_term: 1 leader_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } } }
I20260926 22:42:31.424687 25462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:31.424642 25464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } } }
I20260926 22:42:31.424801 25464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:31.424947 25471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:42:31.425654 25471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:42:31.427661 25471 catalog_manager.cc:1347] Generated new cluster ID: e44bfc9786f04412a025629331412cc1
I20260926 22:42:31.427721 25471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:42:31.427878 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.416928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.11.107.254:44595)
0926 22:42:31.417111 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:31.417121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:31.417154 (+ 33us) server_negotiation.cc:407] Connection header received
0926 22:42:31.417675 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:31.417683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:31.417756 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:31.417928 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:31.418476 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.419833 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.420624 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.421231 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.421340 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.423874 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:31.423879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:31.423879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:31.423892 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:31.426133 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:31.426534 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:31.426537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:31.426538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:31.426635 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:31.426868 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:31.426869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:31.426870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:31.427098 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:31.427257 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:31.427569 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 22:42:31.427698 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260926 22:42:31.452132 25471 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:42:31.452734 25471 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:42:31.459496 25471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Generated new TSK 0
I20260926 22:42:31.459700 25471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:42:31.490669 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44595
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:31.570402 25483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:31.570629 25483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:31.570665 25483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:31.571798 25483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:31.571861 25483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:31.571904 25483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:42:31.573328 25483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:31.573597 25483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:31.573827 25483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:31.578694 25489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:31.578706 25491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:31.578724 25488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:31.578950 25483 server_base.cc:1034] running on GCE node
I20260926 22:42:31.579169 25483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:31.579406 25483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:31.580564 25483 hybrid_clock.cc:648] HybridClock initialized: now 1790462551580544 us; error 29 us; skew 500 ppm
Sep 26 22:42:31 dist-test-slave-j8sq krb5kdc[25376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:31.583007 25483 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:42:31.583467 25483 webserver.cc:460] Webserver started at http://127.11.107.193:38795/ using document root <none> and password file <none>
I20260926 22:42:31.583695 25483 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:31.583762 25483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:31.583916 25483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:42:31.584962 25483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "9236762b010c4d41b10b343be1a5da4a"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "9ae651de5cb7726b047214caf7890cd8"
server_key_iv: "7e558eb74ddd3119ef3813ea784ea602"
server_key_version: "encryptionkey@0"
I20260926 22:42:31.585345 25483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "9236762b010c4d41b10b343be1a5da4a"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "9ae651de5cb7726b047214caf7890cd8"
server_key_iv: "7e558eb74ddd3119ef3813ea784ea602"
server_key_version: "encryptionkey@0"
I20260926 22:42:31.586719 25483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:42:31.587407 25498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:31.587579 25483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:42:31.587684 25483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "9236762b010c4d41b10b343be1a5da4a"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "9ae651de5cb7726b047214caf7890cd8"
server_key_iv: "7e558eb74ddd3119ef3813ea784ea602"
server_key_version: "encryptionkey@0"
I20260926 22:42:31.587757 25483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:31.606418 25483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:31.608125 25483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:31.608285 25483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:31.608505 25483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:31.608803 25483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:31.608855 25483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:31.608893 25483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:31.608922 25483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:31.614496 25483 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:36893
I20260926 22:42:31.614540 25611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:36893 every 8 connection(s)
I20260926 22:42:31.614835 25483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:42:31.615479 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 25483
I20260926 22:42:31.615586 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:42:31.615785 11695 external_mini_cluster.cc:1428] Setting key b0cc7bf4769d58412e583ee0dda326f2
Sep 26 22:42:31 dist-test-slave-j8sq krb5kdc[25376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
I20260926 22:42:31.624294 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.615977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:52101 (local address 127.11.107.254:44595)
0926 22:42:31.616035 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:31.616037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:31.616051 (+ 14us) server_negotiation.cc:407] Connection header received
0926 22:42:31.616259 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:31.616260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:31.616269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:31.616337 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:31.617319 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.617817 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.618894 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.619225 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.619289 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.621415 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:31.621417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:31.621417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:31.621426 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:31.622851 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:31.623273 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:31.623275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:31.623275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:31.623333 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:31.623586 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:31.623587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:31.623587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:31.623723 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:31.623785 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:31.624112 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 22:42:31.624206 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:42:31.624475 25614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.615425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:52101)
0926 22:42:31.615587 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:31.615627 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:31.616175 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:31.616440 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:31.616445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:31.617028 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:31.617207 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:31.617213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.617981 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.617982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:31.618790 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:31.618792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.619317 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.619318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:31.619422 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.619910 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:31.619925 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:31.621328 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:31.622953 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:31.622961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:31.622963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:31.623185 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:31.623403 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:31.623404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:31.623404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:31.623513 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:31.623818 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:31.623823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:31.623966 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 22:42:31.624092 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 22:42:31.624241 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260926 22:42:31.625018 25612 heartbeater.cc:347] Connected to a master server at 127.11.107.254:44595
I20260926 22:42:31.625146 25612 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:31.625316 25612 heartbeater.cc:511] Master 127.11.107.254:44595 requested a full tablet report, sending...
I20260926 22:42:31.625826 25421 ts_manager.cc:194] Registered new tserver with Master: 9236762b010c4d41b10b343be1a5da4a (127.11.107.193:36893)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
I20260926 22:42:31.626500 25421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:52101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:42:31.640374 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44595
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:31.722621 25617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:31.722831 25617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:31.722855 25617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:31.724042 25617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:31.724084 25617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:31.724157 25617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:42:31.725751 25617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:31.726012 25617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:31.726251 25617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:31.731084 25622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:31.731061 25623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:31.731124 25617 server_base.cc:1034] running on GCE node
W20260926 22:42:31.731076 25625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:31.731506 25617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:31.731768 25617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:31.732906 25617 hybrid_clock.cc:648] HybridClock initialized: now 1790462551732894 us; error 33 us; skew 500 ppm
Sep 26 22:42:31 dist-test-slave-j8sq krb5kdc[25376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:31.735396 25617 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:42:31.735752 25617 webserver.cc:460] Webserver started at http://127.11.107.194:38965/ using document root <none> and password file <none>
I20260926 22:42:31.735932 25617 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:31.735971 25617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:31.736056 25617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:42:31.736999 25617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "2535f02c0f9849778c0ab663b7579937"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "979be7d997fd368632b77adf5d2e08ed"
server_key_iv: "d76d585b12c570c68ab730adc69bb11d"
server_key_version: "encryptionkey@0"
I20260926 22:42:31.737277 25617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "2535f02c0f9849778c0ab663b7579937"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "979be7d997fd368632b77adf5d2e08ed"
server_key_iv: "d76d585b12c570c68ab730adc69bb11d"
server_key_version: "encryptionkey@0"
I20260926 22:42:31.738502 25617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:42:31.739174 25632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:31.739346 25617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:42:31.739425 25617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "2535f02c0f9849778c0ab663b7579937"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "979be7d997fd368632b77adf5d2e08ed"
server_key_iv: "d76d585b12c570c68ab730adc69bb11d"
server_key_version: "encryptionkey@0"
I20260926 22:42:31.739477 25617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:31.757372 25617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:31.759085 25617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:31.759186 25617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:31.759343 25617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:31.759564 25617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:31.759593 25617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:31.759613 25617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:31.759626 25617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:31.765254 25617 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:38733
I20260926 22:42:31.765424 25745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:38733 every 8 connection(s)
I20260926 22:42:31.765645 25617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:42:31 dist-test-slave-j8sq krb5kdc[25376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:31.773736 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.766364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:34769 (local address 127.11.107.254:44595)
0926 22:42:31.766436 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:31.766438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:31.766573 (+ 135us) server_negotiation.cc:407] Connection header received
0926 22:42:31.767007 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:31.767008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:31.767018 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:31.767067 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:31.767913 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.768391 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.769289 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.769462 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.769509 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.771460 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:31.771461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:31.771462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:31.771468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:31.772667 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:31.773010 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:31.773011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:31.773011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:31.773054 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:31.773270 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:31.773270 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:31.773271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:31.773361 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:31.773409 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:31.773620 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 22:42:31.773675 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 22:42:31.773922 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.766307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:34769)
0926 22:42:31.766463 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:31.766496 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:31.766930 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:31.767116 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:31.767119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:31.767664 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:31.767842 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:31.767847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.768466 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.768466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:31.769220 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:31.769221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.769531 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.769532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:31.769628 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.770017 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:31.770025 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:31.771379 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:31.772737 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:31.772743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:31.772744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:31.772947 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:31.773112 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:31.773113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:31.773113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:31.773217 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:31.773415 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:31.773419 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:31.773519 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 22:42:31.773595 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 22:42:31.773701 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260926 22:42:31.774425 25746 heartbeater.cc:347] Connected to a master server at 127.11.107.254:44595
I20260926 22:42:31.774529 25746 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:31.774722 25746 heartbeater.cc:511] Master 127.11.107.254:44595 requested a full tablet report, sending...
I20260926 22:42:31.774715 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 25617
I20260926 22:42:31.774823 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:42:31.775053 11695 external_mini_cluster.cc:1428] Setting key bdb1cdf3bdd71cac189d50f5770422c7
I20260926 22:42:31.775117 25421 ts_manager.cc:194] Registered new tserver with Master: 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733)
I20260926 22:42:31.775535 25421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:34769
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:42:31.798962 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44595
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:31.878122 25751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:31.878314 25751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:31.878335 25751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:31.879474 25751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:31.879514 25751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:31.879563 25751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:42:31.880946 25751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:31.881181 25751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:31.881397 25751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:31.886134 25756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:31.886134 25757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:31.886183 25759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:31.886370 25751 server_base.cc:1034] running on GCE node
I20260926 22:42:31.886646 25751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:31.886873 25751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:31.888012 25751 hybrid_clock.cc:648] HybridClock initialized: now 1790462551888002 us; error 26 us; skew 500 ppm
Sep 26 22:42:31 dist-test-slave-j8sq krb5kdc[25376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:31.890714 25751 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:42:31.891170 25751 webserver.cc:460] Webserver started at http://127.11.107.195:40713/ using document root <none> and password file <none>
I20260926 22:42:31.891422 25751 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:31.891490 25751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:31.891650 25751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:42:31.893157 25751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "30e2ca02ef32436182a5efe6d3ec6543"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "8e9f4a9cacd6a2cd059dab97d11bed2a"
server_key_iv: "e11d5255ba2d295ba340cf15d366d026"
server_key_version: "encryptionkey@0"
I20260926 22:42:31.893550 25751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "30e2ca02ef32436182a5efe6d3ec6543"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "8e9f4a9cacd6a2cd059dab97d11bed2a"
server_key_iv: "e11d5255ba2d295ba340cf15d366d026"
server_key_version: "encryptionkey@0"
I20260926 22:42:31.894977 25751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260926 22:42:31.895705 25766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:31.895889 25751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:42:31.895979 25751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "30e2ca02ef32436182a5efe6d3ec6543"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "8e9f4a9cacd6a2cd059dab97d11bed2a"
server_key_iv: "e11d5255ba2d295ba340cf15d366d026"
server_key_version: "encryptionkey@0"
I20260926 22:42:31.896049 25751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:31.908813 25751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:31.910606 25751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:31.910770 25751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:31.910981 25751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:31.911283 25751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:31.911342 25751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:31.911375 25751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:31.911402 25751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:31.916983 25751 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:46415
I20260926 22:42:31.917016 25879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:46415 every 8 connection(s)
I20260926 22:42:31.917320 25751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:42:31 dist-test-slave-j8sq krb5kdc[25376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:31.923403 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 25751
I20260926 22:42:31.923509 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:42:31.923740 11695 external_mini_cluster.cc:1428] Setting key a4b560b686fc88e72fb781bdfb31c700
I20260926 22:42:31.926738 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.918065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:59461 (local address 127.11.107.254:44595)
0926 22:42:31.918142 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:31.918150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:31.918379 (+ 229us) server_negotiation.cc:407] Connection header received
0926 22:42:31.918900 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:31.918902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:31.918914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:31.918961 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:31.919886 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.920407 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.921375 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.921574 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.921646 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.923766 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:31.923768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:31.923768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:31.923775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:31.925254 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:31.925815 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:31.925816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:31.925816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:31.925893 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:31.926201 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:31.926202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:31.926202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:31.926296 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:31.926347 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:31.926618 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 22:42:31.926676 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:42:31.926967 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.918048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:59461)
0926 22:42:31.918242 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:31.918288 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:31.918789 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:31.919028 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:31.919032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:31.919607 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:31.919787 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:31.919792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.920495 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.920496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:31.921264 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:31.921266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.921664 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.921665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:31.921791 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.922274 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:31.922287 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:31.923665 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:31.925344 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:31.925350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:31.925352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:31.925693 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:31.925965 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:31.925966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:31.925966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:31.926120 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:31.926358 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:31.926362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:31.926470 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 22:42:31.926595 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 22:42:31.926714 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
I20260926 22:42:31.927434 25880 heartbeater.cc:347] Connected to a master server at 127.11.107.254:44595
I20260926 22:42:31.927546 25880 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:31.927745 25880 heartbeater.cc:511] Master 127.11.107.254:44595 requested a full tablet report, sending...
I20260926 22:42:31.928148 25421 ts_manager.cc:194] Registered new tserver with Master: 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415)
I20260926 22:42:31.928555 25421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:59461
I20260926 22:42:31.935444 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:42:31.942488 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.936711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.11.107.254:44595)
0926 22:42:31.936762 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:31.936764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:31.936867 (+ 103us) server_negotiation.cc:407] Connection header received
0926 22:42:31.936915 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:31.936915 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:31.936924 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:31.936981 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:31.937535 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.938053 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.938748 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.938915 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.938962 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.940109 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:31.940110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:31.940110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:31.940115 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:31.941375 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:31.941712 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:31.941713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:31.941713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:31.941757 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:31.941991 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:31.941992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:31.941992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:31.942183 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:31.942261 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:31.942377 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 22:42:31.942431 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:42:31.943966 25421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:42:31.945511 25421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:42:31.952045 25891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.947688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:36893 (local address 127.0.0.1:47710)
0926 22:42:31.947993 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:31.948007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:31.948053 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:31.948340 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:31.948341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:31.948344 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:31.948408 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:31.948409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.949718 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.949720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:31.950833 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:31.950836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.951727 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.951728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:31.951819 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.951834 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 22:42:31.951910 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 22:42:31.951966 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":230,"threads_started":1}
I20260926 22:42:31.952250 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.948042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:38733 (local address 127.0.0.1:35880)
0926 22:42:31.948136 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:31.948146 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:31.948191 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:31.948644 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:31.948644 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:31.948648 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:31.948687 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:31.948688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.949553 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.949554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:31.950991 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:31.950992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.951832 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.951833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:31.951890 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.951894 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:42:31.951942 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 22:42:31.951973 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":66,"thread_start_us":42,"threads_started":1}
I20260926 22:42:31.952059 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.947790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.11.107.193:36893)
0926 22:42:31.948168 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:31.948173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:31.948182 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:42:31.948233 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:31.948235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:31.948251 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:31.948332 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:31.948470 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.949593 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.950921 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.951640 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.951719 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.951776 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:31.951936 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 22:42:31.951979 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":78,"threads_started":1}
I20260926 22:42:31.952473 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.948131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.11.107.194:38733)
0926 22:42:31.948336 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:31.948340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:31.948349 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:42:31.948387 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:31.948389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:31.948404 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:31.948575 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:31.948768 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.949442 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.951073 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.951755 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.951829 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.951862 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:31.952015 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 22:42:31.952082 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
I20260926 22:42:31.952946 25546 tablet_service.cc:1467] Processing CreateTablet for tablet 42879799f4f846b6af289685e1014710 (DEFAULT_TABLE table=test-table [id=e9375c7b74f043b6b611b16b2c50dcc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:42:31.953101 25680 tablet_service.cc:1467] Processing CreateTablet for tablet 42879799f4f846b6af289685e1014710 (DEFAULT_TABLE table=test-table [id=e9375c7b74f043b6b611b16b2c50dcc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:42:31.953302 25546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42879799f4f846b6af289685e1014710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:31.953397 25680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42879799f4f846b6af289685e1014710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:31.956542 25895 tablet_bootstrap.cc:492] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Bootstrap starting.
I20260926 22:42:31.956780 25894 tablet_bootstrap.cc:492] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937: Bootstrap starting.
I20260926 22:42:31.957044 25889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.947748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.11.107.195:46415)
0926 22:42:31.947898 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:31.947902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:31.947913 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:42:31.948028 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:31.948031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:31.948046 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:31.948119 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:31.948497 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.949559 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.951913 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.956726 (+ 4813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.956787 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.956835 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:31.956963 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 22:42:31.956991 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260926 22:42:31.957041 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:31.947686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:46415 (local address 127.0.0.1:44526)
0926 22:42:31.947804 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:31.947844 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:31.947907 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:31.948192 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:31.948195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:31.948197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:31.948256 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:31.948258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.949662 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.949663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:31.950989 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:31.950992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:31.956803 (+ 5811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:31.956805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:31.956869 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:31.956873 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:42:31.956946 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 22:42:31.956984 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":112,"thread_start_us":70,"threads_started":1}
I20260926 22:42:31.957688 25814 tablet_service.cc:1467] Processing CreateTablet for tablet 42879799f4f846b6af289685e1014710 (DEFAULT_TABLE table=test-table [id=e9375c7b74f043b6b611b16b2c50dcc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:42:31.958019 25814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42879799f4f846b6af289685e1014710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:31.958065 25895 tablet_bootstrap.cc:654] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:31.958693 25895 log.cc:824] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:31.959353 25894 tablet_bootstrap.cc:654] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:31.959468 25895 tablet_bootstrap.cc:492] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: No bootstrap required, opened a new log
I20260926 22:42:31.959538 25895 ts_tablet_manager.cc:1397] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 22:42:31.959990 25894 log.cc:824] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:31.960743 25894 tablet_bootstrap.cc:492] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937: No bootstrap required, opened a new log
I20260926 22:42:31.960809 25894 ts_tablet_manager.cc:1397] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 22:42:31.961248 25895 raft_consensus.cc:348] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } }
I20260926 22:42:31.961405 25895 raft_consensus.cc:374] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:31.961449 25895 raft_consensus.cc:729] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9236762b010c4d41b10b343be1a5da4a, State: Initialized, Role: FOLLOWER
I20260926 22:42:31.961589 25898 tablet_bootstrap.cc:492] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543: Bootstrap starting.
I20260926 22:42:31.961573 25895 consensus_queue.cc:260] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } }
I20260926 22:42:31.961862 25895 ts_tablet_manager.cc:1428] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:42:31.962010 25612 heartbeater.cc:503] Master 127.11.107.254:44595 was elected leader, sending a full tablet report...
I20260926 22:42:31.962249 25898 tablet_bootstrap.cc:654] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:31.962770 25894 raft_consensus.cc:348] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } }
I20260926 22:42:31.962888 25898 log.cc:824] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:31.962934 25894 raft_consensus.cc:374] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:31.962985 25894 raft_consensus.cc:729] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2535f02c0f9849778c0ab663b7579937, State: Initialized, Role: FOLLOWER
I20260926 22:42:31.963101 25894 consensus_queue.cc:260] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } }
I20260926 22:42:31.963375 25894 ts_tablet_manager.cc:1428] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:42:31.963388 25746 heartbeater.cc:503] Master 127.11.107.254:44595 was elected leader, sending a full tablet report...
I20260926 22:42:31.963644 25898 tablet_bootstrap.cc:492] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543: No bootstrap required, opened a new log
I20260926 22:42:31.963713 25898 ts_tablet_manager.cc:1397] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:42:31.965266 25898 raft_consensus.cc:348] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } }
I20260926 22:42:31.965381 25898 raft_consensus.cc:374] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:31.965402 25898 raft_consensus.cc:729] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e2ca02ef32436182a5efe6d3ec6543, State: Initialized, Role: FOLLOWER
I20260926 22:42:31.965528 25898 consensus_queue.cc:260] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } }
I20260926 22:42:31.965808 25898 ts_tablet_manager.cc:1428] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:42:31.965823 25880 heartbeater.cc:503] Master 127.11.107.254:44595 was elected leader, sending a full tablet report...
W20260926 22:42:32.016203 25747 tablet.cc:2369] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:42:32.115617 25613 tablet.cc:2369] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:42:32.157261 25899 raft_consensus.cc:482] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:42:32.157425 25899 raft_consensus.cc:504] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } }
I20260926 22:42:32.157896 25899 leader_election.cc:288] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415), 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733)
I20260926 22:42:32.161571 25889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.158123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:46097 (local address 127.11.107.195:46415)
0926 22:42:32.158215 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.158218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.158302 (+ 84us) server_negotiation.cc:407] Connection header received
0926 22:42:32.158365 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.158366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.158377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.158428 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:32.158594 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.159430 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.160807 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.161218 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.161304 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.161314 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.161467 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.161505 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 22:42:32.161590 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.158063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:46415 (local address 127.11.107.193:46097)
0926 22:42:32.158236 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:32.158247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:32.158308 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:32.158432 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:32.158433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:32.158436 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:32.158513 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.158515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.159523 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.159524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.160716 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.160718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.161302 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.161303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.161372 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.161376 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:42:32.161459 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 22:42:32.161499 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":58,"threads_started":1}
I20260926 22:42:32.161986 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.158161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:37233 (local address 127.11.107.194:38733)
0926 22:42:32.158254 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.158256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.158892 (+ 636us) server_negotiation.cc:407] Connection header received
0926 22:42:32.158958 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.158959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.158968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.159023 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:32.159176 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.159686 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.160966 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.161354 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.161438 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.161446 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.161833 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.161914 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 22:42:32.162070 25834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42879799f4f846b6af289685e1014710" candidate_uuid: "9236762b010c4d41b10b343be1a5da4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e2ca02ef32436182a5efe6d3ec6543" is_pre_election: true
I20260926 22:42:32.162180 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.158184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:38733 (local address 127.11.107.193:37233)
0926 22:42:32.158825 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:32.158836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:32.158903 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:32.159027 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:32.159028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:32.159029 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:32.159096 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.159098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.159788 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.159789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.160877 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.160880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.161438 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.161439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.161550 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.161554 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:42:32.161605 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:42:32.161642 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":603,"threads_started":1}
I20260926 22:42:32.162256 25834 raft_consensus.cc:2454] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9236762b010c4d41b10b343be1a5da4a in term 0.
I20260926 22:42:32.162470 25700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42879799f4f846b6af289685e1014710" candidate_uuid: "9236762b010c4d41b10b343be1a5da4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2535f02c0f9849778c0ab663b7579937" is_pre_election: true
I20260926 22:42:32.162614 25700 raft_consensus.cc:2454] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9236762b010c4d41b10b343be1a5da4a in term 0.
I20260926 22:42:32.162850 25500 leader_election.cc:302] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e2ca02ef32436182a5efe6d3ec6543, 9236762b010c4d41b10b343be1a5da4a; no voters:
I20260926 22:42:32.163025 25899 raft_consensus.cc:2781] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:42:32.163132 25899 raft_consensus.cc:482] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:42:32.163177 25899 raft_consensus.cc:3037] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:32.163904 25899 raft_consensus.cc:504] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } }
I20260926 22:42:32.164057 25899 leader_election.cc:288] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 1 election: Requested vote from peers 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415), 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733)
I20260926 22:42:32.164225 25700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42879799f4f846b6af289685e1014710" candidate_uuid: "9236762b010c4d41b10b343be1a5da4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2535f02c0f9849778c0ab663b7579937"
I20260926 22:42:32.164233 25834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42879799f4f846b6af289685e1014710" candidate_uuid: "9236762b010c4d41b10b343be1a5da4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e2ca02ef32436182a5efe6d3ec6543"
I20260926 22:42:32.164326 25700 raft_consensus.cc:3037] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:32.164323 25834 raft_consensus.cc:3037] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:32.165024 25700 raft_consensus.cc:2454] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9236762b010c4d41b10b343be1a5da4a in term 1.
I20260926 22:42:32.165063 25834 raft_consensus.cc:2454] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9236762b010c4d41b10b343be1a5da4a in term 1.
I20260926 22:42:32.165220 25500 leader_election.cc:302] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e2ca02ef32436182a5efe6d3ec6543, 9236762b010c4d41b10b343be1a5da4a; no voters:
I20260926 22:42:32.165336 25899 raft_consensus.cc:2781] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:42:32.165488 25899 raft_consensus.cc:686] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 1 LEADER]: Becoming Leader. State: Replica: 9236762b010c4d41b10b343be1a5da4a, State: Running, Role: LEADER
I20260926 22:42:32.165596 25899 consensus_queue.cc:237] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } }
I20260926 22:42:32.166482 25420 catalog_manager.cc:5495] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9236762b010c4d41b10b343be1a5da4a (127.11.107.193). New cstate: current_term: 1 leader_uuid: "9236762b010c4d41b10b343be1a5da4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } health_report { overall_health: UNKNOWN } } }
W20260926 22:42:32.167795 25881 tablet.cc:2369] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:42:32.180962 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.178820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47716 (local address 127.11.107.193:36893)
0926 22:42:32.179009 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.179011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.179019 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:42:32.179048 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.179049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.179065 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.179131 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:42:32.179240 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.179676 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.180342 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.180510 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.180556 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.180673 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:42:32.180762 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:42:32.180819 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.180848 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.180862 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260926 22:42:32.182674 25834 raft_consensus.cc:1261] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543 [term 1 FOLLOWER]: Refusing update from remote peer 9236762b010c4d41b10b343be1a5da4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:42:32.182675 25700 raft_consensus.cc:1261] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937 [term 1 FOLLOWER]: Refusing update from remote peer 9236762b010c4d41b10b343be1a5da4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:42:32.183007 25899 consensus_queue.cc:1035] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:42:32.183132 25905 consensus_queue.cc:1035] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:42:32.185284 25908 mvcc.cc:205] Tried to move back new op lower bound from 7333734613737656320 to 7333734613670842368. Current Snapshot: MvccSnapshot[applied={T|T < 7333734613737656320}]
I20260926 22:42:32.186075 25910 mvcc.cc:205] Tried to move back new op lower bound from 7333734613737656320 to 7333734613670842368. Current Snapshot: MvccSnapshot[applied={T|T < 7333734613737656320 or (T in {7333734613737656320})}]
I20260926 22:42:32.186978 25909 mvcc.cc:205] Tried to move back new op lower bound from 7333734613737656320 to 7333734613670842368. Current Snapshot: MvccSnapshot[applied={T|T < 7333734613737656320 or (T in {7333734613737656320})}]
I20260926 22:42:32.187767 25420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:42:32.187866 25420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:42:32.188603 25420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Sending DeleteTablet for 3 replicas of tablet 42879799f4f846b6af289685e1014710
I20260926 22:42:32.188927 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 25483
I20260926 22:42:32.189035 25546 tablet_service.cc:1514] Processing DeleteTablet for tablet 42879799f4f846b6af289685e1014710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:32 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:47710
I20260926 22:42:32.189534 25680 tablet_service.cc:1514] Processing DeleteTablet for tablet 42879799f4f846b6af289685e1014710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:32 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:35880
I20260926 22:42:32.189749 25917 tablet_replica.cc:330] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937: stopping tablet replica
I20260926 22:42:32.189867 25917 raft_consensus.cc:2229] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:42:32.189949 25814 tablet_service.cc:1514] Processing DeleteTablet for tablet 42879799f4f846b6af289685e1014710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:32 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:44526
I20260926 22:42:32.189977 25917 raft_consensus.cc:2258] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:42:32.190776 25917 ts_tablet_manager.cc:1905] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:42:32.192203 25917 ts_tablet_manager.cc:1918] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:42:32.192302 25917 log.cc:1196] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/42879799f4f846b6af289685e1014710
I20260926 22:42:32.192534 25917 ts_tablet_manager.cc:1939] T 42879799f4f846b6af289685e1014710 P 2535f02c0f9849778c0ab663b7579937: Deleting consensus metadata
I20260926 22:42:32.192926 25408 catalog_manager.cc:4841] TS 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733): tablet 42879799f4f846b6af289685e1014710 (table test-table [id=e9375c7b74f043b6b611b16b2c50dcc5]) successfully deleted
I20260926 22:42:32.193961 25918 tablet_replica.cc:330] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543: stopping tablet replica
I20260926 22:42:32.194067 25918 raft_consensus.cc:2229] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:42:32.194175 25918 raft_consensus.cc:2258] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:42:32.194609 25918 ts_tablet_manager.cc:1905] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 22:42:32.194975 25408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:36893 (error 108)
I20260926 22:42:32.195395 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 25617
I20260926 22:42:32.195454 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.195307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:36893 (local address 127.0.0.1:47724)
0926 22:42:32.195373 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:32.195421 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 22:42:32.196027 25408 catalog_manager.cc:4584] TS 9236762b010c4d41b10b343be1a5da4a (127.11.107.193:36893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42879799f4f846b6af289685e1014710: Network error: Client connection negotiation failed: client connection to 127.11.107.193:36893: connect: Connection refused (error 111)
I20260926 22:42:32.196841 25918 ts_tablet_manager.cc:1918] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:42:32.196939 25918 log.cc:1196] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/42879799f4f846b6af289685e1014710
I20260926 22:42:32.197212 25918 ts_tablet_manager.cc:1939] T 42879799f4f846b6af289685e1014710 P 30e2ca02ef32436182a5efe6d3ec6543: Deleting consensus metadata
I20260926 22:42:32.197676 25406 catalog_manager.cc:4841] TS 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415): tablet 42879799f4f846b6af289685e1014710 (table test-table [id=e9375c7b74f043b6b611b16b2c50dcc5]) successfully deleted
I20260926 22:42:32.201586 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 25751
I20260926 22:42:32.206591 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 25389
I20260926 22:42:32.211928 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:44595
--webserver_interface=127.11.107.254
--webserver_port=41455
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:44595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:32.293957 25920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:32.294152 25920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:32.294174 25920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:32.295351 25920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:42:32.295394 25920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:32.295408 25920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:32.295421 25920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:42:32.295476 25920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:42:32.296891 25920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:44595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:44595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=41455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:32.297137 25920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:32.297365 25920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:32.305661 25927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:32.305634 25929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:32.305650 25926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:32.305703 25920 server_base.cc:1034] running on GCE node
I20260926 22:42:32.306067 25920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:32.306413 25920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:32.307576 25920 hybrid_clock.cc:648] HybridClock initialized: now 1790462552307571 us; error 34 us; skew 500 ppm
Sep 26 22:42:32 dist-test-slave-j8sq krb5kdc[25376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462552, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:32.315415 25920 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:42:32.315804 25920 webserver.cc:460] Webserver started at http://127.11.107.254:41455/ using document root <none> and password file <none>
I20260926 22:42:32.316020 25920 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:32.316078 25920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:32.317466 25920 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:42:32.318193 25936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:32.318390 25920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:42:32.318470 25920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "7c379db711dc4fe6a2f5edeadbd30d69"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "276014c64d4897982bb8e6c3f0cf5e76"
server_key_iv: "fd5f28b92dc40635457098d419980df6"
server_key_version: "encryptionkey@0"
I20260926 22:42:32.318830 25920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:32.339041 25920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:32.341316 25920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:32.341423 25920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:32.345129 25920 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:44595
I20260926 22:42:32.345175 25988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:44595 every 8 connection(s)
I20260926 22:42:32.345535 25920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:42:32.346818 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 25920
I20260926 22:42:32.347134 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:36893
--local_ip_for_outbound_sockets=127.11.107.193
--tserver_master_addrs=127.11.107.254:44595
--webserver_port=38795
--webserver_interface=127.11.107.193
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:42:32.347369 25989 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:42:32.348414 25989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Bootstrap starting.
I20260926 22:42:32.356638 25989 log.cc:824] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:32.358583 25989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:42:32.358873 25989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Bootstrap complete.
I20260926 22:42:32.361184 25989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } }
I20260926 22:42:32.361406 25989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c379db711dc4fe6a2f5edeadbd30d69, State: Initialized, Role: FOLLOWER
I20260926 22:42:32.361563 25989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } }
I20260926 22:42:32.361619 25989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:42:32.361668 25989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:42:32.361718 25989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:42:32.362746 25989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } }
I20260926 22:42:32.362826 25989 leader_election.cc:302] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c379db711dc4fe6a2f5edeadbd30d69; no voters:
I20260926 22:42:32.363030 25989 leader_election.cc:288] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 22:42:32.363072 25993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 22:42:32.363227 25993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 2 LEADER]: Becoming Leader. State: Replica: 7c379db711dc4fe6a2f5edeadbd30d69, State: Running, Role: LEADER
I20260926 22:42:32.363305 25989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:42:32.363373 25993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } }
I20260926 22:42:32.363664 25993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c379db711dc4fe6a2f5edeadbd30d69. Latest consensus state: current_term: 2 leader_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } } }
I20260926 22:42:32.363766 25993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:32.363665 25994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } } }
I20260926 22:42:32.364046 25994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:32.364209 26000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:42:32.364943 26000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:42:32.365036 26000 catalog_manager.cc:1259] Loaded cluster ID: e44bfc9786f04412a025629331412cc1
I20260926 22:42:32.365114 26000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:42:32.365476 26000 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 22:42:32.370296 26000 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 22:42:32.372594 26000 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:42:32.372782 26000 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Loaded TSK: 0
I20260926 22:42:32.372997 26000 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 22:42:32.429265 25991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:32.429484 25991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:32.429520 25991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:32.430662 25991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:32.430722 25991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:32.430766 25991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:42:32.432260 25991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:36893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=38795
--tserver_master_addrs=127.11.107.254:44595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:32.432519 25991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:32.432739 25991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:32.438544 26019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:32.438581 26017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:32.438549 26016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:32.438908 25991 server_base.cc:1034] running on GCE node
I20260926 22:42:32.439138 25991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:32.439383 25991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:32.440544 25991 hybrid_clock.cc:648] HybridClock initialized: now 1790462552440533 us; error 31 us; skew 500 ppm
Sep 26 22:42:32 dist-test-slave-j8sq krb5kdc[25376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462552, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:32.443238 25991 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:42:32.443747 25991 webserver.cc:460] Webserver started at http://127.11.107.193:38795/ using document root <none> and password file <none>
I20260926 22:42:32.443975 25991 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:32.444072 25991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:32.445487 25991 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:32.446283 26026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:32.446461 25991 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 22:42:32.446563 25991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "9236762b010c4d41b10b343be1a5da4a"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "9ae651de5cb7726b047214caf7890cd8"
server_key_iv: "7e558eb74ddd3119ef3813ea784ea602"
server_key_version: "encryptionkey@0"
I20260926 22:42:32.446913 25991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:32.457629 25991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:32.460069 25991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:32.460238 25991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:32.460470 25991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:32.460911 26033 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 22:42:32.462749 25991 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 22:42:32.462822 25991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 22:42:32.462877 25991 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 22:42:32.463646 25991 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 22:42:32.463701 25991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 22:42:32.463749 26033 tablet_bootstrap.cc:492] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Bootstrap starting.
I20260926 22:42:32.470898 25991 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:36893
I20260926 22:42:32.470939 26140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:36893 every 8 connection(s)
I20260926 22:42:32.471386 25991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:42:32.472062 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 25991
I20260926 22:42:32.472379 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:38733
--local_ip_for_outbound_sockets=127.11.107.194
--tserver_master_addrs=127.11.107.254:44595
--webserver_port=38965
--webserver_interface=127.11.107.194
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:42:32.474628 26033 log.cc:824] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:32.476816 26033 tablet_bootstrap.cc:492] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:42:32.477139 26033 tablet_bootstrap.cc:492] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Bootstrap complete.
I20260926 22:42:32.478367 26033 ts_tablet_manager.cc:1397] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260926 22:42:32.479753 26033 raft_consensus.cc:348] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } }
I20260926 22:42:32.480106 26033 raft_consensus.cc:729] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9236762b010c4d41b10b343be1a5da4a, State: Initialized, Role: FOLLOWER
I20260926 22:42:32.480271 26033 consensus_queue.cc:260] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } }
I20260926 22:42:32.480574 26033 ts_tablet_manager.cc:1428] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
Sep 26 22:42:32 dist-test-slave-j8sq krb5kdc[25376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462552, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:32.486737 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.472134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:33277)
0926 22:42:32.474960 (+ 2826us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:32.475002 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:32.475673 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:32.476201 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:32.476206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:32.476568 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:32.476857 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.476863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.478702 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.478703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.479958 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.479960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.480939 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.480940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.481032 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.481617 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:32.481625 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:32.483073 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:32.485007 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:32.485014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:32.485016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:32.485245 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:32.485502 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:32.485502 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:32.485503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:32.485612 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:32.486094 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:32.486099 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:32.486246 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 22:42:32.486357 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 22:42:32.486531 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2784,"thread_start_us":89,"threads_started":1}
I20260926 22:42:32.486749 26144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.472246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:33277 (local address 127.11.107.254:44595)
0926 22:42:32.473993 (+ 1747us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.474004 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.475123 (+ 1119us) server_negotiation.cc:407] Connection header received
0926 22:42:32.475921 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.475928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.475987 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.476143 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:32.477053 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.478595 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.480077 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.480766 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.480841 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.483186 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:32.483191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:32.483191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:32.483202 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:32.484902 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:32.485331 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:32.485334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:32.485334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:32.485430 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:32.485688 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:32.485689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:32.485689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:32.486009 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:32.486115 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.486422 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.486559 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1685,"thread_start_us":118,"threads_started":1}
I20260926 22:42:32.487794 26141 heartbeater.cc:347] Connected to a master server at 127.11.107.254:44595
I20260926 22:42:32.487890 26141 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:32.488063 26141 heartbeater.cc:511] Master 127.11.107.254:44595 requested a full tablet report, sending...
I20260926 22:42:32.488793 25953 ts_manager.cc:194] Registered new tserver with Master: 9236762b010c4d41b10b343be1a5da4a (127.11.107.193:36893)
I20260926 22:42:32.489979 25953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:33277
I20260926 22:42:32.490773 26141 heartbeater.cc:503] Master 127.11.107.254:44595 was elected leader, sending a full tablet report...
I20260926 22:42:32.496783 26148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.489379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:36893 (local address 127.0.0.1:47736)
0926 22:42:32.489968 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:32.490002 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:32.490075 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:32.494127 (+ 4052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:32.494130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:32.494132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:32.494194 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.494196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.495004 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.495005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.495819 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.495822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.496460 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.496461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.496542 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.496555 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 22:42:32.496656 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 22:42:32.496711 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":95,"threads_started":1}
I20260926 22:42:32.496783 26149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.489618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.11.107.193:36893)
0926 22:42:32.493971 (+ 4353us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.493974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.493983 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:42:32.494009 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.494011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.494023 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.494122 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:32.494265 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.494919 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.495921 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.496377 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.496462 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.496515 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.496673 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.496711 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4313,"thread_start_us":69,"threads_started":1}
I20260926 22:42:32.497465 26061 tablet_service.cc:1514] Processing DeleteTablet for tablet 42879799f4f846b6af289685e1014710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:32 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:47736
I20260926 22:42:32.498014 26150 tablet_replica.cc:330] stopping tablet replica
I20260926 22:42:32.498143 26150 raft_consensus.cc:2229] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:42:32.498210 26150 raft_consensus.cc:2258] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:42:32.498503 26150 ts_tablet_manager.cc:1905] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:42:32.500443 26150 ts_tablet_manager.cc:1918] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:42:32.500528 26150 log.cc:1196] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal/wals/42879799f4f846b6af289685e1014710
I20260926 22:42:32.500770 26150 ts_tablet_manager.cc:1939] T 42879799f4f846b6af289685e1014710 P 9236762b010c4d41b10b343be1a5da4a: Deleting consensus metadata
I20260926 22:42:32.501272 25940 catalog_manager.cc:4841] TS 9236762b010c4d41b10b343be1a5da4a (127.11.107.193:36893): tablet 42879799f4f846b6af289685e1014710 (table test-table [id=e9375c7b74f043b6b611b16b2c50dcc5]) successfully deleted
W20260926 22:42:32.564677 26145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:32.564922 26145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:32.564949 26145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:32.566054 26145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:32.566087 26145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:32.566116 26145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:42:32.567451 26145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:38733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=38965
--tserver_master_addrs=127.11.107.254:44595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:32.567619 26145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:32.567788 26145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:32.572547 26155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:32.572544 26158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:32.572738 26145 server_base.cc:1034] running on GCE node
W20260926 22:42:32.572752 26156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:32.573031 26145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:32.573222 26145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:32.574357 26145 hybrid_clock.cc:648] HybridClock initialized: now 1790462552574329 us; error 41 us; skew 500 ppm
Sep 26 22:42:32 dist-test-slave-j8sq krb5kdc[25376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462552, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:32.576901 26145 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:42:32.577332 26145 webserver.cc:460] Webserver started at http://127.11.107.194:38965/ using document root <none> and password file <none>
I20260926 22:42:32.577558 26145 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:32.577641 26145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:32.579396 26145 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:32.580150 26165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:32.580394 26145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:32.580487 26145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "2535f02c0f9849778c0ab663b7579937"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "979be7d997fd368632b77adf5d2e08ed"
server_key_iv: "d76d585b12c570c68ab730adc69bb11d"
server_key_version: "encryptionkey@0"
I20260926 22:42:32.580848 26145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:32.604434 26145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:32.607002 26145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:32.607110 26145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:32.607290 26145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:32.607606 26145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:32.607661 26145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:32.607707 26145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:32.607756 26145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:32.613292 26145 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:38733
I20260926 22:42:32.613365 26278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:38733 every 8 connection(s)
I20260926 22:42:32.613715 26145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:42:32 dist-test-slave-j8sq krb5kdc[25376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462552, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:32.621750 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 26145
I20260926 22:42:32.622031 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:46415
--local_ip_for_outbound_sockets=127.11.107.195
--tserver_master_addrs=127.11.107.254:44595
--webserver_port=40713
--webserver_interface=127.11.107.195
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:42:32.622879 26144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.614448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:47377 (local address 127.11.107.254:44595)
0926 22:42:32.614507 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.614510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.614735 (+ 225us) server_negotiation.cc:407] Connection header received
0926 22:42:32.615264 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.615265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.615281 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.615334 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:32.616245 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.616743 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.617728 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.618038 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.618121 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.620070 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:32.620071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:32.620072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:32.620078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:32.621343 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:32.621850 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:32.621852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:32.621852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:32.621919 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:32.622207 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:32.622208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:32.622208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:32.622345 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:32.622388 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.622696 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.622790 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 22:42:32.623055 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.614448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:47377)
0926 22:42:32.614616 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:32.614651 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:32.615150 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:32.615401 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:32.615405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:32.615957 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:32.616134 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.616138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.616841 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.616841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.617632 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.617634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.618130 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.618131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.618263 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.618681 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:32.618690 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:32.619988 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:32.621416 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:32.621422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:32.621424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:32.621676 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:32.621990 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:32.621991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:32.621991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:32.622099 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:32.622439 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:32.622445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:32.622551 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 22:42:32.622662 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 22:42:32.622810 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260926 22:42:32.623550 26279 heartbeater.cc:347] Connected to a master server at 127.11.107.254:44595
I20260926 22:42:32.623752 26279 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:32.623921 26279 heartbeater.cc:511] Master 127.11.107.254:44595 requested a full tablet report, sending...
I20260926 22:42:32.624347 25953 ts_manager.cc:194] Registered new tserver with Master: 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733)
I20260926 22:42:32.624788 25953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:47377
W20260926 22:42:32.700847 26282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:32.701074 26282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:32.701109 26282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:32.702340 26282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:32.702389 26282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:32.702428 26282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:42:32.703859 26282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:46415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=40713
--tserver_master_addrs=127.11.107.254:44595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:32.704036 26282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:32.704274 26282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:32.709056 26288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:32.709051 26290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:32.709105 26287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:32.709352 26282 server_base.cc:1034] running on GCE node
I20260926 22:42:32.709533 26282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:32.709756 26282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:32.710891 26282 hybrid_clock.cc:648] HybridClock initialized: now 1790462552710888 us; error 34 us; skew 500 ppm
Sep 26 22:42:32 dist-test-slave-j8sq krb5kdc[25376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462552, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:32.713575 26282 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:42:32.714076 26282 webserver.cc:460] Webserver started at http://127.11.107.195:40713/ using document root <none> and password file <none>
I20260926 22:42:32.714318 26282 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:32.714373 26282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:32.715651 26282 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:32.716188 26297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:32.716378 26282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:42:32.716444 26282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "30e2ca02ef32436182a5efe6d3ec6543"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "8e9f4a9cacd6a2cd059dab97d11bed2a"
server_key_iv: "e11d5255ba2d295ba340cf15d366d026"
server_key_version: "encryptionkey@0"
I20260926 22:42:32.716771 26282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:32.728803 26282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:32.730574 26282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:32.730719 26282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:32.730939 26282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:32.731225 26282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:32.731257 26282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:32.731307 26282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:32.731335 26282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:32.737111 26282 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:46415
I20260926 22:42:32.737145 26410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:46415 every 8 connection(s)
I20260926 22:42:32.737471 26282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:42:32 dist-test-slave-j8sq krb5kdc[25376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462552, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:32.745709 26144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.738194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:57567 (local address 127.11.107.254:44595)
0926 22:42:32.738275 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.738278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.738466 (+ 188us) server_negotiation.cc:407] Connection header received
0926 22:42:32.738960 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.738961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.738971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.739032 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:32.739896 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.740372 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.741295 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.741471 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.741519 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.743507 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:32.743508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:32.743509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:32.743514 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:32.744628 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:32.744957 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:32.744958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:32.744958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:32.745002 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:32.745191 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:32.745192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:32.745192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:32.745279 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:32.745325 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.745580 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.745636 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 22:42:32.745882 26413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.738159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:57567)
0926 22:42:32.738327 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:32.738361 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:32.738873 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:32.739082 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:32.739085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:32.739625 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:32.739810 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.739814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.740447 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.740448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.741221 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.741222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.741539 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.741539 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.741639 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.742161 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:32.742169 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:32.743432 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:32.744694 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:32.744699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:32.744700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:32.744896 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:32.745057 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:32.745057 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:32.745057 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:32.745142 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:32.745330 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:32.745334 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:32.745470 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 22:42:32.745554 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 22:42:32.745659 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260926 22:42:32.746356 26411 heartbeater.cc:347] Connected to a master server at 127.11.107.254:44595
I20260926 22:42:32.746429 26411 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:32.746551 26411 heartbeater.cc:511] Master 127.11.107.254:44595 requested a full tablet report, sending...
I20260926 22:42:32.746898 25953 ts_manager.cc:194] Registered new tserver with Master: 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415)
I20260926 22:42:32.747175 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 26282
I20260926 22:42:32.747380 25953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:57567
I20260926 22:42:32.753110 26144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.747567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.11.107.254:44595)
0926 22:42:32.747603 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.747605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.747743 (+ 138us) server_negotiation.cc:407] Connection header received
0926 22:42:32.747832 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.747833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.747842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.747889 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:32.748519 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.749025 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.749682 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.749910 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.749992 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.751002 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:32.751003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:32.751003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:32.751008 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:32.752316 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:32.752598 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:32.752598 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:32.752599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:32.752645 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:32.752790 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:32.752790 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:32.752790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:32.752869 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:32.752981 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.753009 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.753058 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:42:32.753772 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:42:32.755996 26144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.754111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.11.107.254:44595)
0926 22:42:32.754161 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.754162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.754219 (+ 57us) server_negotiation.cc:407] Connection header received
0926 22:42:32.754273 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.754273 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.754297 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.754356 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:42:32.754425 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.754835 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.755486 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.755647 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.755693 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.755807 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:42:32.755883 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:42:32.755934 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.755946 (+ 12us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.755957 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 22:42:32.756322 25953 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:42:32.757911 25953 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:42:32.759980 26061 tablet_service.cc:1467] Processing CreateTablet for tablet c947bfd74d9f4d039400cfacabf93794 (DEFAULT_TABLE table=test-table [id=3d8c6add1a124f26a0099aac65302a84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:42:32.760174 26061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c947bfd74d9f4d039400cfacabf93794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:32.761669 26033 tablet_bootstrap.cc:492] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: Bootstrap starting.
I20260926 22:42:32.761999 26033 tablet_bootstrap.cc:654] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:32.762558 26033 tablet_bootstrap.cc:492] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: No bootstrap required, opened a new log
I20260926 22:42:32.762629 26033 ts_tablet_manager.cc:1397] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:32.762759 26033 raft_consensus.cc:348] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:32.762857 26033 raft_consensus.cc:374] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:32.762902 26033 raft_consensus.cc:729] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9236762b010c4d41b10b343be1a5da4a, State: Initialized, Role: FOLLOWER
I20260926 22:42:32.762944 26033 consensus_queue.cc:260] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:32.763037 26033 ts_tablet_manager.cc:1428] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:32.763545 26148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.759573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:46415 (local address 127.0.0.1:44538)
0926 22:42:32.759605 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:32.759614 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:32.759652 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:32.760048 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:32.760049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:32.760051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:32.760086 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.760087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.761321 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.761322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.762359 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.762363 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.763324 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.763325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.763387 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.763390 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:42:32.763417 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 22:42:32.763462 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 22:42:32.763738 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.759878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.11.107.194:38733)
0926 22:42:32.760042 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.760046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.760056 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:42:32.760100 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.760102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.760118 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.760192 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:32.760398 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.761094 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.762506 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.763221 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.763303 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.763362 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.763522 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.763568 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260926 22:42:32.763952 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.759654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44538 (local address 127.11.107.195:46415)
0926 22:42:32.759886 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.759890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.759899 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:42:32.759930 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.759932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.759946 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.760154 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:32.760202 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.761205 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.762458 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.763236 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.763723 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.763820 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.763845 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.763883 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
I20260926 22:42:32.764005 26416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.759619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:38733 (local address 127.0.0.1:35886)
0926 22:42:32.759719 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:32.759730 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:32.759775 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:32.760293 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:32.760294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:32.760296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:32.760336 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.760337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.761192 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.761194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.762416 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.762419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.763371 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.763373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.763431 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.763436 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:42:32.763495 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 22:42:32.763542 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":39,"thread_start_us":67,"threads_started":1}
I20260926 22:42:32.764657 26213 tablet_service.cc:1467] Processing CreateTablet for tablet c947bfd74d9f4d039400cfacabf93794 (DEFAULT_TABLE table=test-table [id=3d8c6add1a124f26a0099aac65302a84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:42:32.764629 26345 tablet_service.cc:1467] Processing CreateTablet for tablet c947bfd74d9f4d039400cfacabf93794 (DEFAULT_TABLE table=test-table [id=3d8c6add1a124f26a0099aac65302a84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:42:32.764959 26345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c947bfd74d9f4d039400cfacabf93794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:32.764959 26213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c947bfd74d9f4d039400cfacabf93794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:32.768424 26420 tablet_bootstrap.cc:492] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543: Bootstrap starting.
I20260926 22:42:32.768517 26421 tablet_bootstrap.cc:492] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937: Bootstrap starting.
I20260926 22:42:32.769150 26421 tablet_bootstrap.cc:654] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:32.769150 26420 tablet_bootstrap.cc:654] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:32.769749 26420 log.cc:824] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:32.769749 26421 log.cc:824] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:32.770506 26420 tablet_bootstrap.cc:492] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543: No bootstrap required, opened a new log
I20260926 22:42:32.770576 26420 ts_tablet_manager.cc:1397] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:42:32.770673 26421 tablet_bootstrap.cc:492] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937: No bootstrap required, opened a new log
I20260926 22:42:32.770743 26421 ts_tablet_manager.cc:1397] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:42:32.772469 26420 raft_consensus.cc:348] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:32.772634 26420 raft_consensus.cc:374] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:32.772668 26420 raft_consensus.cc:729] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e2ca02ef32436182a5efe6d3ec6543, State: Initialized, Role: FOLLOWER
I20260926 22:42:32.772603 26421 raft_consensus.cc:348] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:32.772754 26421 raft_consensus.cc:374] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:32.772796 26421 raft_consensus.cc:729] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2535f02c0f9849778c0ab663b7579937, State: Initialized, Role: FOLLOWER
I20260926 22:42:32.772789 26420 consensus_queue.cc:260] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:32.772897 26421 consensus_queue.cc:260] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:32.773072 26420 ts_tablet_manager.cc:1428] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:42:32.773087 26411 heartbeater.cc:503] Master 127.11.107.254:44595 was elected leader, sending a full tablet report...
I20260926 22:42:32.773164 26421 ts_tablet_manager.cc:1428] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:42:32.773285 26279 heartbeater.cc:503] Master 127.11.107.254:44595 was elected leader, sending a full tablet report...
W20260926 22:42:32.864264 26280 tablet.cc:2369] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:42:32.943908 26147 raft_consensus.cc:482] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:42:32.944077 26147 raft_consensus.cc:504] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:32.944434 26147 leader_election.cc:288] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415), 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733)
I20260926 22:42:32.947633 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.944708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:47707 (local address 127.11.107.194:38733)
0926 22:42:32.944764 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.944765 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.944898 (+ 133us) server_negotiation.cc:407] Connection header received
0926 22:42:32.944932 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.944933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.944940 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.944981 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:32.945136 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.945819 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.946862 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.947265 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.947324 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.947331 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.947517 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.947560 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:42:32.947639 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.944588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:46415 (local address 127.11.107.193:55873)
0926 22:42:32.944645 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:32.944659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:32.944715 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:32.944872 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:32.944873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:32.944875 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:32.944949 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.944951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.945943 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.945944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.946799 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.946802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.947383 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.947384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.947451 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.947454 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:42:32.947525 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 22:42:32.947563 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":37}
I20260926 22:42:32.947651 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.944668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:55873 (local address 127.11.107.195:46415)
0926 22:42:32.944731 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.944733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.944742 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:42:32.944774 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.944774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.944786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.944844 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:32.945025 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.945789 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.946914 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.947279 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.947373 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.947381 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.947545 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.947587 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:42:32.947654 26426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.944613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:38733 (local address 127.11.107.193:47707)
0926 22:42:32.944798 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:32.944813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:32.944878 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:32.945001 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:32.945001 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:32.945003 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:32.945072 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.945074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.945927 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.945928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.946772 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:32.946775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.947365 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.947366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:32.947444 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.947447 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:42:32.947490 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 22:42:32.947550 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260926 22:42:32.948019 26233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947bfd74d9f4d039400cfacabf93794" candidate_uuid: "9236762b010c4d41b10b343be1a5da4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2535f02c0f9849778c0ab663b7579937" is_pre_election: true
I20260926 22:42:32.948014 26365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947bfd74d9f4d039400cfacabf93794" candidate_uuid: "9236762b010c4d41b10b343be1a5da4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e2ca02ef32436182a5efe6d3ec6543" is_pre_election: true
I20260926 22:42:32.948175 26233 raft_consensus.cc:2454] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9236762b010c4d41b10b343be1a5da4a in term 0.
I20260926 22:42:32.948175 26365 raft_consensus.cc:2454] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9236762b010c4d41b10b343be1a5da4a in term 0.
I20260926 22:42:32.948386 26030 leader_election.cc:302] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2535f02c0f9849778c0ab663b7579937, 9236762b010c4d41b10b343be1a5da4a; no voters:
I20260926 22:42:32.948518 26147 raft_consensus.cc:2781] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:42:32.948570 26147 raft_consensus.cc:482] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:42:32.948597 26147 raft_consensus.cc:3037] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:32.949247 26147 raft_consensus.cc:504] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:32.949381 26147 leader_election.cc:288] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 1 election: Requested vote from peers 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415), 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733)
I20260926 22:42:32.949537 26233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947bfd74d9f4d039400cfacabf93794" candidate_uuid: "9236762b010c4d41b10b343be1a5da4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2535f02c0f9849778c0ab663b7579937"
I20260926 22:42:32.949566 26365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947bfd74d9f4d039400cfacabf93794" candidate_uuid: "9236762b010c4d41b10b343be1a5da4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e2ca02ef32436182a5efe6d3ec6543"
I20260926 22:42:32.949636 26365 raft_consensus.cc:3037] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:32.949635 26233 raft_consensus.cc:3037] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:32.950409 26365 raft_consensus.cc:2454] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9236762b010c4d41b10b343be1a5da4a in term 1.
I20260926 22:42:32.950541 26233 raft_consensus.cc:2454] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9236762b010c4d41b10b343be1a5da4a in term 1.
I20260926 22:42:32.950572 26028 leader_election.cc:302] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e2ca02ef32436182a5efe6d3ec6543, 9236762b010c4d41b10b343be1a5da4a; no voters:
I20260926 22:42:32.950642 26147 raft_consensus.cc:2781] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:42:32.950762 26147 raft_consensus.cc:686] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 LEADER]: Becoming Leader. State: Replica: 9236762b010c4d41b10b343be1a5da4a, State: Running, Role: LEADER
I20260926 22:42:32.950858 26147 consensus_queue.cc:237] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:32.951723 25952 catalog_manager.cc:5495] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9236762b010c4d41b10b343be1a5da4a (127.11.107.193). New cstate: current_term: 1 leader_uuid: "9236762b010c4d41b10b343be1a5da4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } health_report { overall_health: HEALTHY } } }
W20260926 22:42:32.974588 26142 tablet.cc:2369] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:42:32.988054 26412 tablet.cc:2369] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:42:32.989216 26149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:32.986816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.11.107.193:36893)
0926 22:42:32.986880 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:32.986882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:32.986891 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:42:32.986966 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:32.986966 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:32.986982 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:32.987060 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:42:32.987194 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.987652 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.988454 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:32.988648 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:32.988733 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:32.988888 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:42:32.988990 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:42:32.989071 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:32.989097 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 22:42:32.989120 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 22:42:32.990650 26365 raft_consensus.cc:1261] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543 [term 1 FOLLOWER]: Refusing update from remote peer 9236762b010c4d41b10b343be1a5da4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:42:32.990646 26233 raft_consensus.cc:1261] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937 [term 1 FOLLOWER]: Refusing update from remote peer 9236762b010c4d41b10b343be1a5da4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:42:32.990998 26427 consensus_queue.cc:1035] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:42:32.991144 26147 consensus_queue.cc:1035] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:42:32.993902 26431 mvcc.cc:205] Tried to move back new op lower bound from 7333734617047404544 to 7333734616887341056. Current Snapshot: MvccSnapshot[applied={T|T < 7333734617047404544}]
I20260926 22:42:32.995862 25952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:42:32.995949 25952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:42:32.996693 25952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Sending DeleteTablet for 3 replicas of tablet c947bfd74d9f4d039400cfacabf93794
I20260926 22:42:32.996995 26061 tablet_service.cc:1514] Processing DeleteTablet for tablet c947bfd74d9f4d039400cfacabf93794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:32 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:47736
I20260926 22:42:32.996989 26345 tablet_service.cc:1514] Processing DeleteTablet for tablet c947bfd74d9f4d039400cfacabf93794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:32 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:44538
I20260926 22:42:32.997093 26150 tablet_replica.cc:330] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: stopping tablet replica
I20260926 22:42:32.997135 26150 raft_consensus.cc:2229] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:42:32.997207 26150 raft_consensus.cc:2258] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:42:32.997200 26213 tablet_service.cc:1514] Processing DeleteTablet for tablet c947bfd74d9f4d039400cfacabf93794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:32 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:35886
I20260926 22:42:32.997416 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 25991
I20260926 22:42:32.997507 26150 ts_tablet_manager.cc:1905] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:42:32.998136 26439 tablet_replica.cc:330] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543: stopping tablet replica
I20260926 22:42:32.998234 26439 raft_consensus.cc:2229] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:42:32.998327 26439 raft_consensus.cc:2258] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:42:32.998687 26439 ts_tablet_manager.cc:1905] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:42:33.000207 26439 ts_tablet_manager.cc:1918] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:42:33.000303 26439 log.cc:1196] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/c947bfd74d9f4d039400cfacabf93794
I20260926 22:42:33.000553 26439 ts_tablet_manager.cc:1939] T c947bfd74d9f4d039400cfacabf93794 P 30e2ca02ef32436182a5efe6d3ec6543: Deleting consensus metadata
I20260926 22:42:33.000954 25938 catalog_manager.cc:4841] TS 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415): tablet c947bfd74d9f4d039400cfacabf93794 (table test-table [id=3d8c6add1a124f26a0099aac65302a84]) successfully deleted
I20260926 22:42:33.001955 26440 tablet_replica.cc:330] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937: stopping tablet replica
I20260926 22:42:33.002085 26440 raft_consensus.cc:2229] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:42:33.002197 26440 raft_consensus.cc:2258] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:42:33.002588 26440 ts_tablet_manager.cc:1905] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 22:42:33.003659 25940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:36893 (error 108)
I20260926 22:42:33.003895 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 26145
I20260926 22:42:33.003921 26440 ts_tablet_manager.cc:1918] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:42:33.003999 26440 log.cc:1196] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/c947bfd74d9f4d039400cfacabf93794
I20260926 22:42:33.004249 26440 ts_tablet_manager.cc:1939] T c947bfd74d9f4d039400cfacabf93794 P 2535f02c0f9849778c0ab663b7579937: Deleting consensus metadata
I20260926 22:42:33.005939 26416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:33.005776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:36893 (local address 127.0.0.1:47758)
0926 22:42:33.005855 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:33.005898 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 22:42:33.006057 25940 catalog_manager.cc:4584] TS 9236762b010c4d41b10b343be1a5da4a (127.11.107.193:36893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c947bfd74d9f4d039400cfacabf93794: Network error: Client connection negotiation failed: client connection to 127.11.107.193:36893: connect: Connection refused (error 111)
I20260926 22:42:33.009193 26416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:33.009099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:38733 (local address 127.0.0.1:35902)
0926 22:42:33.009138 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:33.009162 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.194:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 22:42:33.009218 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 26282
I20260926 22:42:33.014008 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 25920
I20260926 22:42:33.019181 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:44595
--webserver_interface=127.11.107.254
--webserver_port=41455
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:44595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:33.098505 26442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:33.098748 26442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:33.098788 26442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:33.099905 26442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:42:33.099975 26442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:33.099999 26442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:33.100021 26442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:42:33.100049 26442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:42:33.101493 26442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:44595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:44595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=41455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:33.101778 26442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:33.102029 26442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:33.106685 26447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:33.106757 26450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:33.106678 26448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:33.106935 26442 server_base.cc:1034] running on GCE node
I20260926 22:42:33.107178 26442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:33.107475 26442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:33.108615 26442 hybrid_clock.cc:648] HybridClock initialized: now 1790462553108597 us; error 31 us; skew 500 ppm
Sep 26 22:42:33 dist-test-slave-j8sq krb5kdc[25376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462553, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:33.111155 26442 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:42:33.111606 26442 webserver.cc:460] Webserver started at http://127.11.107.254:41455/ using document root <none> and password file <none>
I20260926 22:42:33.111838 26442 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:33.111883 26442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:33.113236 26442 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:33.113886 26457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:33.114065 26442 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:42:33.114161 26442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "7c379db711dc4fe6a2f5edeadbd30d69"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "276014c64d4897982bb8e6c3f0cf5e76"
server_key_iv: "fd5f28b92dc40635457098d419980df6"
server_key_version: "encryptionkey@0"
I20260926 22:42:33.114492 26442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:33.153672 26442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:33.156006 26442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:33.156169 26442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:33.160282 26442 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:44595
I20260926 22:42:33.160327 26509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:44595 every 8 connection(s)
I20260926 22:42:33.160741 26442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:42:33.162334 26510 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:42:33.162945 26510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Bootstrap starting.
I20260926 22:42:33.163836 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 26442
I20260926 22:42:33.164089 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:36893
--local_ip_for_outbound_sockets=127.11.107.193
--tserver_master_addrs=127.11.107.254:44595
--webserver_port=38795
--webserver_interface=127.11.107.193
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:42:33.171460 26510 log.cc:824] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:33.173996 26510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:42:33.174319 26510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69: Bootstrap complete.
I20260926 22:42:33.176522 26510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } }
I20260926 22:42:33.176772 26510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c379db711dc4fe6a2f5edeadbd30d69, State: Initialized, Role: FOLLOWER
I20260926 22:42:33.176986 26510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } }
I20260926 22:42:33.177098 26510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:42:33.177152 26510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:42:33.177202 26510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 2 FOLLOWER]: Advancing to term 3
I20260926 22:42:33.178274 26510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } }
I20260926 22:42:33.178395 26510 leader_election.cc:302] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c379db711dc4fe6a2f5edeadbd30d69; no voters:
I20260926 22:42:33.178601 26510 leader_election.cc:288] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 22:42:33.178638 26514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 22:42:33.178750 26514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [term 3 LEADER]: Becoming Leader. State: Replica: 7c379db711dc4fe6a2f5edeadbd30d69, State: Running, Role: LEADER
I20260926 22:42:33.178854 26514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } }
I20260926 22:42:33.178882 26510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:42:33.179147 26515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } } }
I20260926 22:42:33.179145 26514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c379db711dc4fe6a2f5edeadbd30d69. Latest consensus state: current_term: 3 leader_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c379db711dc4fe6a2f5edeadbd30d69" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44595 } } }
I20260926 22:42:33.179267 26515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:33.179271 26514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c379db711dc4fe6a2f5edeadbd30d69 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:33.179451 26518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:42:33.180311 26518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:42:33.180439 26518 catalog_manager.cc:1259] Loaded cluster ID: e44bfc9786f04412a025629331412cc1
I20260926 22:42:33.180691 26518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 22:42:33.181023 26518 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb6718fa9ad google::LogMessage::Fail() at ??:0
@ 0x7fb6718fd88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb6718fa4de google::LogMessage::Flush() at ??:0
@ 0x7fb6718fba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb67981b0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb6798248b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb6737c80d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb6737bd954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb6746216db start_thread at ??:0
@ 0x7fb6703f861f clone at ??:0
W20260926 22:42:33.276892 26512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:33.277184 26512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:33.277242 26512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:33.278903 26512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:33.278961 26512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:33.278998 26512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:42:33.281152 26512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:36893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=38795
--tserver_master_addrs=127.11.107.254:44595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:33.281430 26512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:33.281692 26512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:33.287046 26549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:33.287168 26547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:33.287039 26546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:33.287335 26512 server_base.cc:1034] running on GCE node
I20260926 22:42:33.287523 26512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:33.287796 26512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:33.288980 26512 hybrid_clock.cc:648] HybridClock initialized: now 1790462553288966 us; error 26 us; skew 500 ppm
Sep 26 22:42:33 dist-test-slave-j8sq krb5kdc[25376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462553, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:33.291904 26512 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:42:33.292332 26512 webserver.cc:460] Webserver started at http://127.11.107.193:38795/ using document root <none> and password file <none>
I20260926 22:42:33.292569 26512 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:33.292639 26512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:33.294035 26512 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:33.294829 26556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:33.295006 26512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:33.295100 26512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "9236762b010c4d41b10b343be1a5da4a"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "9ae651de5cb7726b047214caf7890cd8"
server_key_iv: "7e558eb74ddd3119ef3813ea784ea602"
server_key_version: "encryptionkey@0"
I20260926 22:42:33.295439 26512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:33.318218 26512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:33.320559 26512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:33.320724 26512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:33.320953 26512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:33.321410 26563 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 22:42:33.323323 26512 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 22:42:33.323382 26512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 22:42:33.323431 26512 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 22:42:33.324025 26512 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 22:42:33.324080 26512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 22:42:33.324167 26563 tablet_bootstrap.cc:492] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: Bootstrap starting.
I20260926 22:42:33.326004 26563 log.cc:824] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:33.329399 26563 tablet_bootstrap.cc:492] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:42:33.329777 26563 tablet_bootstrap.cc:492] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: Bootstrap complete.
I20260926 22:42:33.329974 26563 ts_tablet_manager.cc:1397] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260926 22:42:33.331030 26512 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:36893
I20260926 22:42:33.331259 26671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:36893 every 8 connection(s)
I20260926 22:42:33.331398 26512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:42:33.332134 26563 raft_consensus.cc:348] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:33.332509 26563 raft_consensus.cc:729] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9236762b010c4d41b10b343be1a5da4a, State: Initialized, Role: FOLLOWER
I20260926 22:42:33.332667 26563 consensus_queue.cc:260] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:33.332955 26563 ts_tablet_manager.cc:1428] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:42:33.339190 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 26512
I20260926 22:42:33.339433 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:38733
--local_ip_for_outbound_sockets=127.11.107.194
--tserver_master_addrs=127.11.107.254:44595
--webserver_port=38965
--webserver_interface=127.11.107.194
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:33.395999 26674 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:42:33.332053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:58345)
0926 22:42:33.332251 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:33.332309 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:33.332454 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:33.395801 (+ 63347us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
W20260926 22:42:33.396325 26672 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:44595 (0 consecutive failures): Network error: Failed to ping master at 127.11.107.254:44595: Client connection negotiation failed: client connection to 127.11.107.254:44595: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 22:42:33.396699 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:33.396562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:50775)
0926 22:42:33.396620 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:33.396668 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 22:42:33.397025 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:33.396937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:47931)
0926 22:42:33.396981 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:33.397007 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 22:42:33.397140 26672 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 22:42:33.419588 26676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:33.419773 26676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:33.419796 26676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:33.420921 26676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:33.420960 26676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:33.421010 26676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:42:33.422426 26676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:38733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=38965
--tserver_master_addrs=127.11.107.254:44595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:33.422669 26676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:33.422886 26676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:33.427650 26681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:33.427692 26682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:33.427788 26676 server_base.cc:1034] running on GCE node
W20260926 22:42:33.427822 26684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:33.428113 26676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:33.428350 26676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:33.429519 26676 hybrid_clock.cc:648] HybridClock initialized: now 1790462553429510 us; error 39 us; skew 500 ppm
Sep 26 22:42:33 dist-test-slave-j8sq krb5kdc[25376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462553, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:33.432085 26676 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:42:33.432510 26676 webserver.cc:460] Webserver started at http://127.11.107.194:38965/ using document root <none> and password file <none>
I20260926 22:42:33.432682 26676 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:33.432722 26676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:33.433960 26676 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:33.434501 26691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:33.434648 26676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:33.434724 26676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "2535f02c0f9849778c0ab663b7579937"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "979be7d997fd368632b77adf5d2e08ed"
server_key_iv: "d76d585b12c570c68ab730adc69bb11d"
server_key_version: "encryptionkey@0"
I20260926 22:42:33.435017 26676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:33.443660 26676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:33.445429 26676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:33.445526 26676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:33.445673 26676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:33.445962 26676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:33.445996 26676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:33.446017 26676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:33.446029 26676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:33.451469 26676 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:38733
I20260926 22:42:33.451542 26804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:38733 every 8 connection(s)
I20260926 22:42:33.451862 26676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:42:33.452849 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:33.452500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:42069)
0926 22:42:33.452647 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:33.452722 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
W20260926 22:42:33.453253 26805 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:44595 (0 consecutive failures): Network error: Failed to ping master at 127.11.107.254:44595: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
I20260926 22:42:33.453526 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:33.453432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:59761)
0926 22:42:33.453481 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:33.453505 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 22:42:33.453789 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:33.453708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:53191)
0926 22:42:33.453750 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:33.453771 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 22:42:33.453941 26805 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 22:42:33.454149 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 26676
I20260926 22:42:33.454427 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:46415
--local_ip_for_outbound_sockets=127.11.107.195
--tserver_master_addrs=127.11.107.254:44595
--webserver_port=40713
--webserver_interface=127.11.107.195
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:33.532840 26808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:33.533072 26808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:33.533109 26808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:33.534286 26808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:33.534355 26808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:33.534401 26808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:42:33.535921 26808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:46415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=40713
--tserver_master_addrs=127.11.107.254:44595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:33.536193 26808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:33.536422 26808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:33.541281 26814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:33.541288 26813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:33.541288 26816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:33.541498 26808 server_base.cc:1034] running on GCE node
I20260926 22:42:33.541716 26808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:33.541985 26808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:33.543154 26808 hybrid_clock.cc:648] HybridClock initialized: now 1790462553543146 us; error 39 us; skew 500 ppm
Sep 26 22:42:33 dist-test-slave-j8sq krb5kdc[25376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462553, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:33.545446 26808 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:42:33.545929 26808 webserver.cc:460] Webserver started at http://127.11.107.195:40713/ using document root <none> and password file <none>
I20260926 22:42:33.546149 26808 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:33.546214 26808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:33.547478 26808 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:42:33.548051 26823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:33.548209 26808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:42:33.548302 26808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "30e2ca02ef32436182a5efe6d3ec6543"
format_stamp: "Formatted at 2026-09-26 22:42:31 on dist-test-slave-j8sq"
server_key: "8e9f4a9cacd6a2cd059dab97d11bed2a"
server_key_iv: "e11d5255ba2d295ba340cf15d366d026"
server_key_version: "encryptionkey@0"
I20260926 22:42:33.548766 26808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:33.575248 26808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:33.577268 26808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:33.577442 26808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:33.577702 26808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:33.578078 26808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:33.578133 26808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:33.578176 26808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:33.578204 26808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:33.584137 26808 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:46415
I20260926 22:42:33.584185 26936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:46415 every 8 connection(s)
I20260926 22:42:33.584527 26808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:42:33.585575 26939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:33.585190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:41717)
0926 22:42:33.585360 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:33.585438 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
W20260926 22:42:33.586050 26937 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:44595 (0 consecutive failures): Network error: Failed to ping master at 127.11.107.254:44595: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
I20260926 22:42:33.586324 26939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:33.586231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:55437)
0926 22:42:33.586281 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:33.586307 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 22:42:33.586588 26939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:33.586512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:34633)
0926 22:42:33.586554 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:33.586574 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 22:42:33.586712 26937 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 22:42:33.588969 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 26808
I20260926 22:42:34.397691 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:34.397474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:54963)
0926 22:42:34.397591 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:34.397624 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260926 22:42:34.454437 26942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:34.454245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:37239)
0926 22:42:34.454361 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:34.454390 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260926 22:42:34.587224 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:34.587006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:46657)
0926 22:42:34.587139 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:34.587176 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":54,"threads_started":1}
I20260926 22:42:34.698295 26944 raft_consensus.cc:482] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:42:34.698417 26944 raft_consensus.cc:504] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:34.698809 26944 leader_election.cc:288] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415), 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733)
Sep 26 22:42:34 dist-test-slave-j8sq krb5kdc[25376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462553, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.194@KRBTEST.COM
Sep 26 22:42:34 dist-test-slave-j8sq krb5kdc[25376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462553, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.195@KRBTEST.COM
I20260926 22:42:34.710902 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:34.698944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:46415 (local address 127.11.107.193:46813)
0926 22:42:34.699066 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:34.699078 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:34.699137 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:34.700152 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:34.700154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:34.700485 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:34.700867 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:34.700869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:34.702370 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:34.702371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:34.703731 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:34.703732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:34.704546 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:34.704548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:34.704697 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:34.705322 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:34.705325 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:34.707425 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:34.709290 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:34.709290 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:34.709291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:34.709584 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:34.709877 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:34.709878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:34.709878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:34.709989 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:34.710394 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:34.710395 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:34.710513 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 22:42:34.710626 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 22:42:34.710812 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":76}
I20260926 22:42:34.710918 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:34.699113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:46813 (local address 127.11.107.195:46415)
0926 22:42:34.699287 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:34.699295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:34.699321 (+ 26us) server_negotiation.cc:407] Connection header received
0926 22:42:34.699894 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:34.699897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:34.699957 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:34.700104 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:34.700964 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:34.702279 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:34.703808 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:34.704439 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:34.704551 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:34.707524 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:34.707528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:34.707529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:34.707542 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:34.709206 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:34.709652 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:34.709655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:34.709655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:34.709773 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:34.710070 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:34.710071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:34.710071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:34.710312 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:34.710400 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:34.710681 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 22:42:34.710801 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260926 22:42:34.711113 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:34.698953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:38733 (local address 127.11.107.193:60949)
0926 22:42:34.699071 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:34.699082 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:34.699136 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:34.700231 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:34.700232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:34.700473 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:34.700698 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:34.700704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:34.702368 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:34.702369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:34.703717 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:34.703721 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:34.704548 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:34.704548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:34.704697 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:34.705307 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:34.705316 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:34.707105 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:34.709124 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:34.709130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:34.709132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:34.709530 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:34.709905 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:34.709906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:34.709907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:34.709994 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:34.710378 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:34.710384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:34.710512 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 22:42:34.710625 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 22:42:34.710886 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":215,"thread_start_us":46,"threads_started":1}
I20260926 22:42:34.710947 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:34.699088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:60949 (local address 127.11.107.194:38733)
0926 22:42:34.699268 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:34.699280 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:34.699320 (+ 40us) server_negotiation.cc:407] Connection header received
0926 22:42:34.699898 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:34.699901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:34.699957 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:34.700104 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:34.700815 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:34.702256 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:34.703808 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:34.704439 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:34.704551 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:34.707225 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:34.707230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:34.707230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:34.707243 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:34.709040 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:34.709612 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:34.709615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:34.709615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:34.709823 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:34.710070 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:34.710071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:34.710072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:34.710297 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:34.710385 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:34.710691 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 22:42:34.710837 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
W20260926 22:42:34.711802 26558 leader_election.cc:341] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415): Not found: Tablet not found: c947bfd74d9f4d039400cfacabf93794
W20260926 22:42:34.712144 26560 leader_election.cc:341] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733): Not found: Tablet not found: c947bfd74d9f4d039400cfacabf93794
I20260926 22:42:34.712193 26560 leader_election.cc:302] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9236762b010c4d41b10b343be1a5da4a; no voters: 2535f02c0f9849778c0ab663b7579937, 30e2ca02ef32436182a5efe6d3ec6543
I20260926 22:42:34.712348 26944 raft_consensus.cc:2726] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:42:35.398422 26948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:35.398189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:45183)
0926 22:42:35.398332 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:35.398374 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260926 22:42:35.454994 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:35.454822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:59625)
0926 22:42:35.454930 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:35.454960 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260926 22:42:35.587919 26950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:35.587715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:40447)
0926 22:42:35.587843 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:35.587877 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260926 22:42:36.399020 26951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:36.398814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:34421)
0926 22:42:36.398941 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:36.398976 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260926 22:42:36.455564 26952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:36.455376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:59353)
0926 22:42:36.455492 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:36.455526 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260926 22:42:36.588632 26953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:36.588412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:52607)
0926 22:42:36.588537 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:36.588589 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260926 22:42:36.861426 26954 raft_consensus.cc:482] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:42:36.861517 26954 raft_consensus.cc:504] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:36.862042 26954 leader_election.cc:288] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415), 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733)
W20260926 22:42:36.862376 26558 leader_election.cc:341] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415): Not found: Tablet not found: c947bfd74d9f4d039400cfacabf93794
W20260926 22:42:36.862449 26560 leader_election.cc:341] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733): Not found: Tablet not found: c947bfd74d9f4d039400cfacabf93794
I20260926 22:42:36.862506 26560 leader_election.cc:302] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9236762b010c4d41b10b343be1a5da4a; no voters: 2535f02c0f9849778c0ab663b7579937, 30e2ca02ef32436182a5efe6d3ec6543
I20260926 22:42:36.862579 26954 raft_consensus.cc:2726] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:42:37.399643 26955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:37.399450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:42451)
0926 22:42:37.399566 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:37.399601 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260926 22:42:37.456121 26956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:37.455938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:53787)
0926 22:42:37.456049 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:37.456084 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260926 22:42:37.589340 26957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:37.589125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:34027)
0926 22:42:37.589268 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:37.589300 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260926 22:42:38.400290 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:38.400069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:58249)
0926 22:42:38.400211 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:38.400246 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260926 22:42:38.456673 26959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:38.456494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:52853)
0926 22:42:38.456604 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:38.456637 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260926 22:42:38.590068 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:38.589758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:38599)
0926 22:42:38.589984 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:38.590023 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":57,"threads_started":1}
I20260926 22:42:38.639112 26961 raft_consensus.cc:482] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:42:38.639248 26961 raft_consensus.cc:504] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:38.639411 26961 leader_election.cc:288] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415), 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733)
W20260926 22:42:38.639793 26558 leader_election.cc:341] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415): Not found: Tablet not found: c947bfd74d9f4d039400cfacabf93794
W20260926 22:42:38.639873 26560 leader_election.cc:341] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733): Not found: Tablet not found: c947bfd74d9f4d039400cfacabf93794
I20260926 22:42:38.639942 26560 leader_election.cc:302] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9236762b010c4d41b10b343be1a5da4a; no voters: 2535f02c0f9849778c0ab663b7579937, 30e2ca02ef32436182a5efe6d3ec6543
I20260926 22:42:38.640026 26961 raft_consensus.cc:2726] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:42:39.400884 26962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:39.400695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:52927)
0926 22:42:39.400805 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:39.400840 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260926 22:42:39.457262 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:39.457080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:50623)
0926 22:42:39.457190 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:39.457223 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260926 22:42:39.590701 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:39.590521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:39519)
0926 22:42:39.590628 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:39.590661 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260926 22:42:40.401502 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:40.401286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:49947)
0926 22:42:40.401413 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:40.401459 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260926 22:42:40.457892 26966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:40.457656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:40141)
0926 22:42:40.457781 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:40.457810 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260926 22:42:40.591346 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:40.591137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:56601)
0926 22:42:40.591259 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:40.591300 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260926 22:42:41.402101 26968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:41.401919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:40497)
0926 22:42:41.402023 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:41.402057 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260926 22:42:41.458461 26969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:41.458266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:33665)
0926 22:42:41.458378 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:41.458411 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260926 22:42:41.591951 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:41.591752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:42915)
0926 22:42:41.591879 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:41.591910 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260926 22:42:42.402673 26971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:42.402468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:57289)
0926 22:42:42.402589 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:42.402629 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260926 22:42:42.459056 26972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:42.458864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:46337)
0926 22:42:42.458980 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:42.459014 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260926 22:42:42.592612 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:42.592383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:33443)
0926 22:42:42.592530 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:42.592567 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":59,"threads_started":1}
I20260926 22:42:43.037597 26974 raft_consensus.cc:482] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:42:43.037689 26974 raft_consensus.cc:504] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:43.037911 26974 leader_election.cc:288] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415), 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733)
W20260926 22:42:43.038230 26558 leader_election.cc:341] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415): Not found: Tablet not found: c947bfd74d9f4d039400cfacabf93794
W20260926 22:42:43.038311 26560 leader_election.cc:341] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733): Not found: Tablet not found: c947bfd74d9f4d039400cfacabf93794
I20260926 22:42:43.038384 26560 leader_election.cc:302] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9236762b010c4d41b10b343be1a5da4a; no voters: 2535f02c0f9849778c0ab663b7579937, 30e2ca02ef32436182a5efe6d3ec6543
I20260926 22:42:43.038470 26974 raft_consensus.cc:2726] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:42:43.403277 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:43.403071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:48125)
0926 22:42:43.403196 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:43.403231 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260926 22:42:43.459649 26976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:43.459463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:46321)
0926 22:42:43.459576 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:43.459611 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260926 22:42:43.593233 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:43.593027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:34123)
0926 22:42:43.593154 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:43.593189 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260926 22:42:44.404011 26978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:44.403814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:50303)
0926 22:42:44.403929 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:44.403967 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260926 22:42:44.460343 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:44.460156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:44503)
0926 22:42:44.460272 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:44.460303 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260926 22:42:44.593900 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:44.593640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:35479)
0926 22:42:44.593772 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:44.593805 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260926 22:42:45.404590 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:45.404404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:43147)
0926 22:42:45.404514 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:45.404547 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260926 22:42:45.460960 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:45.460764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:50011)
0926 22:42:45.460878 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:45.460919 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260926 22:42:45.594497 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:45.594307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:46337)
0926 22:42:45.594418 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:45.594452 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260926 22:42:46.405246 26984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:46.405032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:58255)
0926 22:42:46.405161 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:46.405201 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260926 22:42:46.461596 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:46.461387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:49975)
0926 22:42:46.461519 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:46.461552 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260926 22:42:46.595216 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:46.594954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:56199)
0926 22:42:46.595124 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:46.595166 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
I20260926 22:42:46.703557 26987 raft_consensus.cc:482] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:42:46.703646 26987 raft_consensus.cc:504] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30e2ca02ef32436182a5efe6d3ec6543" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 46415 } } peers { permanent_uuid: "2535f02c0f9849778c0ab663b7579937" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 38733 } } peers { permanent_uuid: "9236762b010c4d41b10b343be1a5da4a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36893 } }
I20260926 22:42:46.703868 26987 leader_election.cc:288] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415), 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733)
W20260926 22:42:46.704231 26560 leader_election.cc:341] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2535f02c0f9849778c0ab663b7579937 (127.11.107.194:38733): Not found: Tablet not found: c947bfd74d9f4d039400cfacabf93794
W20260926 22:42:46.704311 26558 leader_election.cc:341] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30e2ca02ef32436182a5efe6d3ec6543 (127.11.107.195:46415): Not found: Tablet not found: c947bfd74d9f4d039400cfacabf93794
I20260926 22:42:46.704370 26558 leader_election.cc:302] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9236762b010c4d41b10b343be1a5da4a; no voters: 2535f02c0f9849778c0ab663b7579937, 30e2ca02ef32436182a5efe6d3ec6543
I20260926 22:42:46.704444 26987 raft_consensus.cc:2726] T c947bfd74d9f4d039400cfacabf93794 P 9236762b010c4d41b10b343be1a5da4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:42:47.406054 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:47.405663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:47601)
0926 22:42:47.405819 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:47.406007 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260926 22:42:47.462253 26989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:47.462062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:41347)
0926 22:42:47.462177 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:47.462211 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260926 22:42:47.596118 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:47.595873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:60449)
0926 22:42:47.596028 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:47.596071 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260926 22:42:48.406755 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:48.406556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.193:39841)
0926 22:42:48.406680 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:48.406710 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260926 22:42:48.462922 26992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:48.462724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.194:52621)
0926 22:42:48.462851 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:48.462883 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260926 22:42:48.594122 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 26512
I20260926 22:42:48.596853 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:48.596658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44595 (local address 127.11.107.195:50691)
0926 22:42:48.596776 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:48.596810 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260926 22:42:48.599085 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 26676
I20260926 22:42:48.603790 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 26808
2026-09-26T22:42:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18451 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:42:48 dist-test-slave-j8sq krb5kdc[26996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:42:48 dist-test-slave-j8sq krb5kdc[26996](info): set up 2 sockets
Sep 26 22:42:48 dist-test-slave-j8sq krb5kdc[26996](info): commencing operation
krb5kdc: starting...
W20260926 22:42:49.634910 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:42:49 dist-test-slave-j8sq krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:42:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:42:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:42:49.748699 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:45429
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:45429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:49.827551 27009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:49.827776 27009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:49.827811 27009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:49.828919 27009 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 22:42:49.828982 27009 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 22:42:49.829003 27009 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 22:42:49.829021 27009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:42:49.829051 27009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:49.829077 27009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:49.829102 27009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:42:49.829133 27009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:42:49.830595 27009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:45429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:45429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:49.830866 27009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:49.831084 27009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:49.836086 27017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:49.836122 27014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:49.836086 27015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:49.836117 27009 server_base.cc:1034] running on GCE node
I20260926 22:42:49.836560 27009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:49.836843 27009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:49.838007 27009 hybrid_clock.cc:648] HybridClock initialized: now 1790462569837994 us; error 25 us; skew 500 ppm
Sep 26 22:42:49 dist-test-slave-j8sq krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462569, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:49.840431 27009 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:42:49.840849 27009 webserver.cc:460] Webserver started at http://127.11.107.254:46161/ using document root <none> and password file <none>
I20260926 22:42:49.841064 27009 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:49.841107 27009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:49.841261 27009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:42:49.842375 27009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "5aaee8f0076547e582f3d5bc7cbcf109"
format_stamp: "Formatted at 2026-09-26 22:42:49 on dist-test-slave-j8sq"
server_key: "ae34ba4d6744ba0aca1e89fb02feddee"
server_key_iv: "d488b20c15fd66637896f5ce9bb68aed"
server_key_version: "encryptionkey@0"
I20260926 22:42:49.842779 27009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "5aaee8f0076547e582f3d5bc7cbcf109"
format_stamp: "Formatted at 2026-09-26 22:42:49 on dist-test-slave-j8sq"
server_key: "ae34ba4d6744ba0aca1e89fb02feddee"
server_key_iv: "d488b20c15fd66637896f5ce9bb68aed"
server_key_version: "encryptionkey@0"
I20260926 22:42:49.844112 27009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:42:49.844830 27024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:49.845016 27009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:42:49.845129 27009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "5aaee8f0076547e582f3d5bc7cbcf109"
format_stamp: "Formatted at 2026-09-26 22:42:49 on dist-test-slave-j8sq"
server_key: "ae34ba4d6744ba0aca1e89fb02feddee"
server_key_iv: "d488b20c15fd66637896f5ce9bb68aed"
server_key_version: "encryptionkey@0"
I20260926 22:42:49.845198 27009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:49.858347 27009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:49.860509 27009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:49.860662 27009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:49.864512 27009 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:45429
I20260926 22:42:49.864579 27076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:45429 every 8 connection(s)
I20260926 22:42:49.864902 27009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:42:49.865773 27077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:49.868057 27077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Bootstrap starting.
I20260926 22:42:49.868635 27077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:49.868942 27077 log.cc:824] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:49.869577 27077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: No bootstrap required, opened a new log
I20260926 22:42:49.871212 27077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } }
I20260926 22:42:49.871366 27077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:49.871411 27077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aaee8f0076547e582f3d5bc7cbcf109, State: Initialized, Role: FOLLOWER
I20260926 22:42:49.871595 27077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } }
I20260926 22:42:49.871701 27077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:42:49.871743 27077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:42:49.871795 27077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:49.872334 27077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } }
I20260926 22:42:49.872449 27077 leader_election.cc:302] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aaee8f0076547e582f3d5bc7cbcf109; no voters:
I20260926 22:42:49.872632 27077 leader_election.cc:288] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:42:49.872675 27080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:42:49.872854 27080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 1 LEADER]: Becoming Leader. State: Replica: 5aaee8f0076547e582f3d5bc7cbcf109, State: Running, Role: LEADER
I20260926 22:42:49.872896 27077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:42:49.872989 27080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } }
I20260926 22:42:49.873322 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 27009
I20260926 22:42:49.873414 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:42:49.873615 11695 external_mini_cluster.cc:1428] Setting key 841e90674d6e9020e034a3d128d4f7c4
I20260926 22:42:49.874162 27082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aaee8f0076547e582f3d5bc7cbcf109. Latest consensus state: current_term: 1 leader_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } } }
I20260926 22:42:49.874264 27082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:49.874562 27096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:42:49.874442 27081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } } }
I20260926 22:42:49.874775 27081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: This master's current role is: LEADER
I20260926 22:42:49.875145 27096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:42:49.876652 27096 catalog_manager.cc:1347] Generated new cluster ID: f27c5eae9f424fbdb3f97e2dae3d0b42
I20260926 22:42:49.876703 27096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:42:49 dist-test-slave-j8sq krb5kdc[26996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:49.885004 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:49.874245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.11.107.254:45429)
0926 22:42:49.874521 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:49.874531 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:49.874564 (+ 33us) server_negotiation.cc:407] Connection header received
0926 22:42:49.874678 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:49.874696 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:49.874780 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:49.874932 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:49.875665 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:49.877012 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:49.878059 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:49.878670 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:49.878794 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:49.881513 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:49.881517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:49.881517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:49.881528 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:49.883424 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:49.883835 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:49.883837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:49.883837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:49.883931 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:49.884137 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:49.884138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:49.884138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:49.884422 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:49.884546 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:49.884679 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 22:42:49.884799 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":59,"threads_started":1}
I20260926 22:42:49.908968 27096 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:42:49.909514 27096 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:42:49.916236 27096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Generated new TSK 0
I20260926 22:42:49.916419 27096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:42:49.946554 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:45429
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:50.025933 27103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:50.026129 27103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:50.026152 27103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:50.027300 27103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:50.027343 27103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:50.027395 27103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:42:50.028820 27103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:50.029058 27103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:50.029291 27103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:50.034303 27108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:50.034423 27103 server_base.cc:1034] running on GCE node
W20260926 22:42:50.034327 27111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:50.034514 27109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:50.034727 27103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:50.034989 27103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:50.036155 27103 hybrid_clock.cc:648] HybridClock initialized: now 1790462570036146 us; error 37 us; skew 500 ppm
Sep 26 22:42:50 dist-test-slave-j8sq krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462570, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:50.038539 27103 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:42:50.038960 27103 webserver.cc:460] Webserver started at http://127.11.107.193:42561/ using document root <none> and password file <none>
I20260926 22:42:50.039181 27103 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:50.039250 27103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:50.039395 27103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:42:50.040408 27103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "12a5559fe9574cdeaecea34832279b54"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "e1345e8b75511a1dd6d60e269f62dba8"
server_key_iv: "585ccdb3cb026d9169457b656366c835"
server_key_version: "encryptionkey@0"
I20260926 22:42:50.040786 27103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "12a5559fe9574cdeaecea34832279b54"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "e1345e8b75511a1dd6d60e269f62dba8"
server_key_iv: "585ccdb3cb026d9169457b656366c835"
server_key_version: "encryptionkey@0"
I20260926 22:42:50.042106 27103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:42:50.042801 27118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:50.042971 27103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:42:50.043067 27103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "12a5559fe9574cdeaecea34832279b54"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "e1345e8b75511a1dd6d60e269f62dba8"
server_key_iv: "585ccdb3cb026d9169457b656366c835"
server_key_version: "encryptionkey@0"
I20260926 22:42:50.043143 27103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:50.075981 27103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:50.077917 27103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:50.078088 27103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:50.078341 27103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:50.078662 27103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:50.078717 27103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:50.078754 27103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:50.078786 27103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:50.084461 27103 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:43167
I20260926 22:42:50.084510 27231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:43167 every 8 connection(s)
I20260926 22:42:50.084823 27103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
Sep 26 22:42:50 dist-test-slave-j8sq krb5kdc[26996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462570, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:50.091486 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 27103
I20260926 22:42:50.091609 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:42:50.091882 11695 external_mini_cluster.cc:1428] Setting key cb1e74a15f7b3037fcfc240cb548f182
I20260926 22:42:50.094633 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.085512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:42863 (local address 127.11.107.254:45429)
0926 22:42:50.085573 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:50.085575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:50.085746 (+ 171us) server_negotiation.cc:407] Connection header received
0926 22:42:50.086299 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:50.086299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:50.086312 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:50.086371 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:50.087392 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.087888 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.088819 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.089014 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.089065 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.091158 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:50.091159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:50.091159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:50.091168 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:50.093122 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:50.093514 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:50.093516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:50.093516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:50.093595 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:50.093836 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:50.093863 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:50.093863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:50.094019 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:50.094099 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:50.094478 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 22:42:50.094557 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:42:50.094856 27234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.085452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:42863)
0926 22:42:50.085617 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:50.085660 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:50.086212 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:50.086445 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:50.086449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:50.087098 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:50.087304 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.087309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.087963 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.087964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.088746 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.088748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.089082 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.089083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.089192 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.089702 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:50.089710 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:50.091045 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:50.093215 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:50.093221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:50.093223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:50.093439 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:50.093669 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:50.093670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:50.093670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:50.093767 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:50.094105 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:50.094110 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:50.094303 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 22:42:50.094427 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 22:42:50.094602 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260926 22:42:50.095359 27232 heartbeater.cc:347] Connected to a master server at 127.11.107.254:45429
I20260926 22:42:50.095486 27232 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:50.095669 27232 heartbeater.cc:511] Master 127.11.107.254:45429 requested a full tablet report, sending...
I20260926 22:42:50.096225 27041 ts_manager.cc:194] Registered new tserver with Master: 12a5559fe9574cdeaecea34832279b54 (127.11.107.193:43167)
I20260926 22:42:50.097182 27041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:42863
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:42:50.117012 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:45429
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:50.195878 27237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:50.196069 27237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:50.196092 27237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:50.197222 27237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:50.197263 27237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:50.197289 27237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:42:50.198762 27237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:50.198992 27237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:50.199222 27237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:42:50.204046 27245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:50.204087 27242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:50.204135 27243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:50.204380 27237 server_base.cc:1034] running on GCE node
I20260926 22:42:50.204573 27237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:50.204852 27237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:50.206009 27237 hybrid_clock.cc:648] HybridClock initialized: now 1790462570205988 us; error 34 us; skew 500 ppm
Sep 26 22:42:50 dist-test-slave-j8sq krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462570, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:50.208293 27237 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:42:50.208698 27237 webserver.cc:460] Webserver started at http://127.11.107.194:40617/ using document root <none> and password file <none>
I20260926 22:42:50.208943 27237 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:50.209009 27237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:50.209139 27237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:42:50.210201 27237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "477ee41f35fa4b91a2361be5bbeb098b"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "ecf0530fd07eee960b04b19a01004e53"
server_key_iv: "5316a1c1c2d77f648c13189f5ae30d93"
server_key_version: "encryptionkey@0"
I20260926 22:42:50.210551 27237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "477ee41f35fa4b91a2361be5bbeb098b"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "ecf0530fd07eee960b04b19a01004e53"
server_key_iv: "5316a1c1c2d77f648c13189f5ae30d93"
server_key_version: "encryptionkey@0"
I20260926 22:42:50.211840 27237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:42:50.212509 27252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:50.212674 27237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:42:50.212762 27237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "477ee41f35fa4b91a2361be5bbeb098b"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "ecf0530fd07eee960b04b19a01004e53"
server_key_iv: "5316a1c1c2d77f648c13189f5ae30d93"
server_key_version: "encryptionkey@0"
I20260926 22:42:50.212841 27237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:50.226747 27237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:50.228452 27237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:50.228610 27237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:50.228837 27237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:50.229135 27237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:50.229188 27237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:50.229223 27237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:50.229252 27237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:50.235108 27237 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:34173
I20260926 22:42:50.235157 27365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:34173 every 8 connection(s)
I20260926 22:42:50.235438 27237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:42:50 dist-test-slave-j8sq krb5kdc[26996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462570, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:50.241379 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 27237
I20260926 22:42:50.241503 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:42:50.241745 11695 external_mini_cluster.cc:1428] Setting key c6da7925fa54c4bc212e9bb02b2a6479
I20260926 22:42:50.245199 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.236082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:38951 (local address 127.11.107.254:45429)
0926 22:42:50.236157 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:50.236160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:50.236329 (+ 169us) server_negotiation.cc:407] Connection header received
0926 22:42:50.236850 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:50.236851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:50.236864 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:50.236914 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:50.237810 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.238353 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.239416 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.239639 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.239714 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.241641 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:50.241643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:50.241643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:50.241652 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:50.243301 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:50.243937 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:50.243939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:50.243939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:50.244005 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:50.244356 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:50.244357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:50.244358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:50.244494 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:50.244636 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:50.245004 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 22:42:50.245111 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 22:42:50.245368 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.236041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:38951)
0926 22:42:50.236209 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:50.236255 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:50.236744 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:50.236974 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:50.236978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:50.237531 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:50.237706 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.237719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.238449 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.238450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.239313 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.239315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.239727 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.239728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.239847 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.240241 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:50.240248 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:50.241548 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:50.243397 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:50.243405 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:50.243407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:50.243782 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:50.244129 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:50.244130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:50.244131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:50.244241 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:50.244629 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:50.244635 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:50.244841 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 22:42:50.244981 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 22:42:50.245132 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260926 22:42:50.245831 27366 heartbeater.cc:347] Connected to a master server at 127.11.107.254:45429
I20260926 22:42:50.245941 27366 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:50.246100 27366 heartbeater.cc:511] Master 127.11.107.254:45429 requested a full tablet report, sending...
I20260926 22:42:50.246449 27041 ts_manager.cc:194] Registered new tserver with Master: 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173)
I20260926 22:42:50.246925 27041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:38951
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:42:50.266750 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:45429
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:42:50.347656 27371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:42:50.347908 27371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:42:50.347934 27371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:42:50.349135 27371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:42:50.349174 27371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:42:50.349200 27371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:42:50.350660 27371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:42:50.350901 27371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:42:50.351145 27371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:42:50.356027 27371 server_base.cc:1034] running on GCE node
W20260926 22:42:50.356000 27376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:50.355995 27377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:42:50.356040 27379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:42:50.356382 27371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:42:50.356650 27371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:42:50.357815 27371 hybrid_clock.cc:648] HybridClock initialized: now 1790462570357809 us; error 31 us; skew 500 ppm
Sep 26 22:42:50 dist-test-slave-j8sq krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462570, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:42:50.360267 27371 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:42:50.360724 27371 webserver.cc:460] Webserver started at http://127.11.107.195:44915/ using document root <none> and password file <none>
I20260926 22:42:50.360961 27371 fs_manager.cc:359] Metadata directory not provided
I20260926 22:42:50.361027 27371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:42:50.361161 27371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:42:50.362264 27371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "91f3775fd6e747dc806b4fe0b8e8ec72"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "abd7553aaf70ef6b544e55b8389d91f6"
server_key_iv: "e294e061091d1ef94fe5b3f48ad40d4b"
server_key_version: "encryptionkey@0"
I20260926 22:42:50.362607 27371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "91f3775fd6e747dc806b4fe0b8e8ec72"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "abd7553aaf70ef6b544e55b8389d91f6"
server_key_iv: "e294e061091d1ef94fe5b3f48ad40d4b"
server_key_version: "encryptionkey@0"
I20260926 22:42:50.363899 27371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:42:50.364634 27386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:50.364816 27371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:42:50.364910 27371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "91f3775fd6e747dc806b4fe0b8e8ec72"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "abd7553aaf70ef6b544e55b8389d91f6"
server_key_iv: "e294e061091d1ef94fe5b3f48ad40d4b"
server_key_version: "encryptionkey@0"
I20260926 22:42:50.364995 27371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:42:50.384209 27371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:42:50.386288 27371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:42:50.386462 27371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:42:50.386704 27371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:42:50.387066 27371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:42:50.387147 27371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:50.387198 27371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:42:50.387225 27371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:42:50.393569 27371 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:33841
I20260926 22:42:50.393625 27499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:33841 every 8 connection(s)
I20260926 22:42:50.393989 27371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:42:50 dist-test-slave-j8sq krb5kdc[26996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462570, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:42:50.401158 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 27371
I20260926 22:42:50.401264 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:42:50.401474 11695 external_mini_cluster.cc:1428] Setting key 81fd7f10855ac5417e647f9212b7bbdc
I20260926 22:42:50.402894 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.394774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:53447 (local address 127.11.107.254:45429)
0926 22:42:50.394833 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:50.394835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:50.395200 (+ 365us) server_negotiation.cc:407] Connection header received
0926 22:42:50.395520 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:50.395521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:50.395532 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:50.395590 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:50.396458 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.396937 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.397895 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.398069 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.398118 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.400127 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:50.400129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:50.400129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:50.400135 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:50.401412 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:50.401868 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:50.401870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:50.401870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:50.401929 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:50.402259 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:50.402260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:50.402260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:50.402379 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:50.402456 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:50.402721 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 22:42:50.402800 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 22:42:50.403093 27502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.394767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:53447)
0926 22:42:50.394910 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:50.394943 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:50.395434 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:50.395650 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:50.395654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:50.396200 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:42:50.396384 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.396388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.397007 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.397008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.397796 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.397798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.398149 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.398149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.398271 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.398787 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:42:50.398795 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:42:50.400045 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:42:50.401482 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:50.401488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:50.401490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:50.401745 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:50.402031 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:42:50.402032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:42:50.402033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:42:50.402176 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:42:50.402447 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:42:50.402453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:42:50.402590 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 22:42:50.402699 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 22:42:50.402861 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260926 22:42:50.403468 27500 heartbeater.cc:347] Connected to a master server at 127.11.107.254:45429
I20260926 22:42:50.403541 27500 heartbeater.cc:464] Registering TS with master...
I20260926 22:42:50.403728 27500 heartbeater.cc:511] Master 127.11.107.254:45429 requested a full tablet report, sending...
I20260926 22:42:50.404099 27041 ts_manager.cc:194] Registered new tserver with Master: 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841)
I20260926 22:42:50.404470 27041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:53447
I20260926 22:42:50.413113 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:42:50.420040 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.414413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.11.107.254:45429)
0926 22:42:50.414470 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:50.414472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:50.414581 (+ 109us) server_negotiation.cc:407] Connection header received
0926 22:42:50.414619 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:50.414620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:50.414628 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:50.414679 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:42:50.415160 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.415666 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.416582 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.416839 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.416895 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.418053 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:42:50.418053 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:42:50.418054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:42:50.418059 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:42:50.419177 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:50.419462 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:50.419463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:50.419463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:50.419509 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:42:50.419620 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:42:50.419621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:42:50.419621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:42:50.419741 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:42:50.419811 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:50.419903 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 22:42:50.419957 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:42:50.421542 27041 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:42:50.423133 27041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:42:50.429463 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.425450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:33841 (local address 127.0.0.1:53546)
0926 22:42:50.425721 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:50.425733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:50.425801 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:50.426116 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:50.426119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:50.426120 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:50.426165 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.426166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.427169 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.427170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.428294 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.428297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.429126 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.429127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.429222 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.429235 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 22:42:50.429325 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 22:42:50.429376 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260926 22:42:50.429656 27512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.425451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:34173 (local address 127.0.0.1:47476)
0926 22:42:50.425922 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:50.425931 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:50.425968 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:50.426346 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:50.426347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:50.426349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:50.426402 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.426404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.427651 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.427652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.428496 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.428498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.429320 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.429321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.429373 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.429376 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:42:50.429429 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 22:42:50.429459 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":7,"thread_start_us":88,"threads_started":1}
I20260926 22:42:50.429730 27514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.425538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.11.107.194:34173)
0926 22:42:50.426112 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:50.426116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:50.426125 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:42:50.426160 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:50.426162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:50.426175 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:50.426264 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:50.426487 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.427545 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.428577 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.429236 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.429312 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.429359 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:50.429453 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 22:42:50.429491 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":84,"threads_started":1}
I20260926 22:42:50.429791 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.425668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.11.107.195:33841)
0926 22:42:50.425888 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:50.425893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:50.425905 (+ 12us) server_negotiation.cc:407] Connection header received
0926 22:42:50.425951 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:50.425953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:50.425970 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:50.426042 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:50.426279 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.427069 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.428398 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.429043 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.429128 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.429183 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:50.429650 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 22:42:50.429700 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
I20260926 22:42:50.429956 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.425451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:43167 (local address 127.0.0.1:38822)
0926 22:42:50.425681 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:50.425732 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:50.425806 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:50.426220 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:50.426221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:50.426223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:50.426260 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.426262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.427633 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.427634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.428723 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.428726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.429670 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.429671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.429737 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.429742 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:42:50.429795 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 22:42:50.429864 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":24,"thread_start_us":40,"threads_started":1}
I20260926 22:42:50.430205 27516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.425877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.11.107.193:43167)
0926 22:42:50.426047 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:50.426052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:50.426063 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:42:50.426104 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:50.426106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:50.426123 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:50.426211 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:50.426404 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.427522 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.428804 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.429523 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.429588 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.429638 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:50.430066 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 22:42:50.430118 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260926 22:42:50.430426 27434 tablet_service.cc:1467] Processing CreateTablet for tablet 3be6188cbbae4da9ab17b37979543c9b (DEFAULT_TABLE table=test-table [id=53563fe41884460ba345ef4478789143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:42:50.430569 27300 tablet_service.cc:1467] Processing CreateTablet for tablet 3be6188cbbae4da9ab17b37979543c9b (DEFAULT_TABLE table=test-table [id=53563fe41884460ba345ef4478789143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:42:50.430806 27434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be6188cbbae4da9ab17b37979543c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:50.430840 27300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be6188cbbae4da9ab17b37979543c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:50.434275 27517 tablet_bootstrap.cc:492] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72: Bootstrap starting.
I20260926 22:42:50.434661 27166 tablet_service.cc:1467] Processing CreateTablet for tablet 3be6188cbbae4da9ab17b37979543c9b (DEFAULT_TABLE table=test-table [id=53563fe41884460ba345ef4478789143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:42:50.434911 27517 tablet_bootstrap.cc:654] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:50.434963 27166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be6188cbbae4da9ab17b37979543c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:42:50.435400 27517 log.cc:824] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:50.435760 27518 tablet_bootstrap.cc:492] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b: Bootstrap starting.
I20260926 22:42:50.436301 27518 tablet_bootstrap.cc:654] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:50.436847 27518 log.cc:824] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:50.437422 27517 tablet_bootstrap.cc:492] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72: No bootstrap required, opened a new log
I20260926 22:42:50.437506 27517 ts_tablet_manager.cc:1397] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 22:42:50.437808 27521 tablet_bootstrap.cc:492] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Bootstrap starting.
I20260926 22:42:50.438335 27521 tablet_bootstrap.cc:654] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Neither blocks nor log segments found. Creating new log.
I20260926 22:42:50.438778 27521 log.cc:824] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Log is configured to *not* fsync() on all Append() calls
I20260926 22:42:50.439092 27517 raft_consensus.cc:348] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:42:50.439227 27517 raft_consensus.cc:374] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:50.439278 27517 raft_consensus.cc:729] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f3775fd6e747dc806b4fe0b8e8ec72, State: Initialized, Role: FOLLOWER
I20260926 22:42:50.439393 27517 consensus_queue.cc:260] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:42:50.439471 27521 tablet_bootstrap.cc:492] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: No bootstrap required, opened a new log
I20260926 22:42:50.439533 27521 ts_tablet_manager.cc:1397] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:42:50.439672 27500 heartbeater.cc:503] Master 127.11.107.254:45429 was elected leader, sending a full tablet report...
I20260926 22:42:50.439672 27517 ts_tablet_manager.cc:1428] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:42:50.439985 27518 tablet_bootstrap.cc:492] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b: No bootstrap required, opened a new log
I20260926 22:42:50.440066 27518 ts_tablet_manager.cc:1397] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 22:42:50.441519 27521 raft_consensus.cc:348] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:42:50.441672 27521 raft_consensus.cc:374] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:50.441720 27521 raft_consensus.cc:729] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a5559fe9574cdeaecea34832279b54, State: Initialized, Role: FOLLOWER
I20260926 22:42:50.441829 27521 consensus_queue.cc:260] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:42:50.442054 27518 raft_consensus.cc:348] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:42:50.442168 27521 ts_tablet_manager.cc:1428] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 22:42:50.442221 27232 heartbeater.cc:503] Master 127.11.107.254:45429 was elected leader, sending a full tablet report...
I20260926 22:42:50.442224 27518 raft_consensus.cc:374] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:42:50.442269 27518 raft_consensus.cc:729] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477ee41f35fa4b91a2361be5bbeb098b, State: Initialized, Role: FOLLOWER
I20260926 22:42:50.442379 27518 consensus_queue.cc:260] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:42:50.442657 27518 ts_tablet_manager.cc:1428] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:42:50.442688 27366 heartbeater.cc:503] Master 127.11.107.254:45429 was elected leader, sending a full tablet report...
W20260926 22:42:50.486073 27367 tablet.cc:2369] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:42:50.504608 27523 raft_consensus.cc:482] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:42:50.504772 27523 raft_consensus.cc:504] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:42:50.505141 27523 leader_election.cc:288] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173), 12a5559fe9574cdeaecea34832279b54 (127.11.107.193:43167)
I20260926 22:42:50.507983 27502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.505317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:34173 (local address 127.11.107.195:57595)
0926 22:42:50.505379 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:50.505391 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:50.505435 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:50.505577 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:50.505578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:50.505596 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:50.505670 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.505672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.506371 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.506372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.507137 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.507139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.507729 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.507731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.507814 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.507817 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:42:50.507873 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 22:42:50.507909 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 22:42:50.508006 27514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.505390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:57595 (local address 127.11.107.194:34173)
0926 22:42:50.505433 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:50.505435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:50.505446 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:42:50.505483 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:50.505484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:50.505494 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:50.505567 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:50.505747 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.506293 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.507230 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.507642 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.507708 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.507715 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:50.507888 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 22:42:50.507928 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:42:50.508100 27526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.505319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:43167 (local address 127.11.107.195:41459)
0926 22:42:50.505533 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:50.505544 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:42:50.505603 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:42:50.505746 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:42:50.505746 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:42:50.505748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:42:50.505816 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.505817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.506495 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.506496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.507259 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:42:50.507262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.507832 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.507833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:42:50.507916 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.507920 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:42:50.507976 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 22:42:50.508009 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":19,"thread_start_us":138,"threads_started":1}
I20260926 22:42:50.508134 27516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.505391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:41459 (local address 127.11.107.193:43167)
0926 22:42:50.505433 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:50.505436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:50.505616 (+ 180us) server_negotiation.cc:407] Connection header received
0926 22:42:50.505649 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:50.505650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:50.505661 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:50.505723 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:42:50.505910 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.506409 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.507351 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.507749 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.507820 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.507828 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:50.507998 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 22:42:50.508041 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:42:50.508437 27186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be6188cbbae4da9ab17b37979543c9b" candidate_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a5559fe9574cdeaecea34832279b54" is_pre_election: true
I20260926 22:42:50.508466 27320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be6188cbbae4da9ab17b37979543c9b" candidate_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477ee41f35fa4b91a2361be5bbeb098b" is_pre_election: true
I20260926 22:42:50.508590 27186 raft_consensus.cc:2454] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91f3775fd6e747dc806b4fe0b8e8ec72 in term 0.
I20260926 22:42:50.508606 27320 raft_consensus.cc:2454] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91f3775fd6e747dc806b4fe0b8e8ec72 in term 0.
I20260926 22:42:50.508839 27387 leader_election.cc:302] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a5559fe9574cdeaecea34832279b54, 91f3775fd6e747dc806b4fe0b8e8ec72; no voters:
I20260926 22:42:50.508983 27523 raft_consensus.cc:2781] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:42:50.509038 27523 raft_consensus.cc:482] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:42:50.509068 27523 raft_consensus.cc:3037] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:50.509729 27523 raft_consensus.cc:504] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:42:50.509897 27523 leader_election.cc:288] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [CANDIDATE]: Term 1 election: Requested vote from peers 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173), 12a5559fe9574cdeaecea34832279b54 (127.11.107.193:43167)
I20260926 22:42:50.510005 27320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be6188cbbae4da9ab17b37979543c9b" candidate_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477ee41f35fa4b91a2361be5bbeb098b"
I20260926 22:42:50.510010 27186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be6188cbbae4da9ab17b37979543c9b" candidate_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a5559fe9574cdeaecea34832279b54"
I20260926 22:42:50.510092 27320 raft_consensus.cc:3037] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:50.510090 27186 raft_consensus.cc:3037] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:42:50.510780 27320 raft_consensus.cc:2454] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f3775fd6e747dc806b4fe0b8e8ec72 in term 1.
I20260926 22:42:50.510782 27186 raft_consensus.cc:2454] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f3775fd6e747dc806b4fe0b8e8ec72 in term 1.
I20260926 22:42:50.510955 27388 leader_election.cc:302] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 477ee41f35fa4b91a2361be5bbeb098b, 91f3775fd6e747dc806b4fe0b8e8ec72; no voters:
I20260926 22:42:50.511053 27523 raft_consensus.cc:2781] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:42:50.511194 27523 raft_consensus.cc:686] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 1 LEADER]: Becoming Leader. State: Replica: 91f3775fd6e747dc806b4fe0b8e8ec72, State: Running, Role: LEADER
I20260926 22:42:50.511289 27523 consensus_queue.cc:237] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:42:50.512125 27040 catalog_manager.cc:5495] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195). New cstate: current_term: 1 leader_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } health_report { overall_health: UNKNOWN } } }
I20260926 22:42:50.520104 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.518199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.11.107.195:33841)
0926 22:42:50.518238 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:42:50.518240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:42:50.518246 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:42:50.518271 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:42:50.518272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:42:50.518285 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:42:50.518343 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:42:50.518432 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.518858 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.519544 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:42:50.519713 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:42:50.519761 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:42:50.519877 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:42:50.519962 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:42:50.520014 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:42:50.520043 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 22:42:50.520057 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:42:50.521690 27320 raft_consensus.cc:1261] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b [term 1 FOLLOWER]: Refusing update from remote peer 91f3775fd6e747dc806b4fe0b8e8ec72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:42:50.521700 27186 raft_consensus.cc:1261] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Refusing update from remote peer 91f3775fd6e747dc806b4fe0b8e8ec72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:42:50.522091 27527 consensus_queue.cc:1035] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:42:50.522224 27523 consensus_queue.cc:1035] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:42:50.524392 27530 mvcc.cc:205] Tried to move back new op lower bound from 7333734688854396928 to 7333734688814882816. Current Snapshot: MvccSnapshot[applied={T|T < 7333734688854396928}]
I20260926 22:42:50.525270 27532 mvcc.cc:205] Tried to move back new op lower bound from 7333734688854396928 to 7333734688814882816. Current Snapshot: MvccSnapshot[applied={T|T < 7333734688854396928}]
I20260926 22:42:50.526486 27041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:42:50.526582 27041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:42:50.527249 27041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Sending DeleteTablet for 3 replicas of tablet 3be6188cbbae4da9ab17b37979543c9b
I20260926 22:42:50.527544 27434 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be6188cbbae4da9ab17b37979543c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:50 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:53546
I20260926 22:42:50.527585 27300 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be6188cbbae4da9ab17b37979543c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:50 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:47476
I20260926 22:42:50.527596 27166 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be6188cbbae4da9ab17b37979543c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:50 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:38822
I20260926 22:42:50.527631 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 27103
I20260926 22:42:50.527776 27540 tablet_replica.cc:330] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72: stopping tablet replica
I20260926 22:42:50.527855 27540 raft_consensus.cc:2229] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:42:50.527930 27540 raft_consensus.cc:2258] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:42:50.528404 27540 ts_tablet_manager.cc:1905] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:42:50.530077 27540 ts_tablet_manager.cc:1918] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:42:50.530153 27540 log.cc:1196] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/3be6188cbbae4da9ab17b37979543c9b
I20260926 22:42:50.530452 27540 ts_tablet_manager.cc:1939] T 3be6188cbbae4da9ab17b37979543c9b P 91f3775fd6e747dc806b4fe0b8e8ec72: Deleting consensus metadata
I20260926 22:42:50.530813 27541 tablet_replica.cc:330] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b: stopping tablet replica
I20260926 22:42:50.530845 27027 catalog_manager.cc:4841] TS 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841): tablet 3be6188cbbae4da9ab17b37979543c9b (table test-table [id=53563fe41884460ba345ef4478789143]) successfully deleted
I20260926 22:42:50.530946 27541 raft_consensus.cc:2229] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:42:50.531039 27541 raft_consensus.cc:2258] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:42:50.531512 27541 ts_tablet_manager.cc:1905] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:42:50.533061 27541 ts_tablet_manager.cc:1918] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:42:50.533134 27541 log.cc:1196] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/3be6188cbbae4da9ab17b37979543c9b
I20260926 22:42:50.533422 27541 ts_tablet_manager.cc:1939] T 3be6188cbbae4da9ab17b37979543c9b P 477ee41f35fa4b91a2361be5bbeb098b: Deleting consensus metadata
W20260926 22:42:50.533725 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:43167 (error 108)
I20260926 22:42:50.533982 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 27237
I20260926 22:42:50.534550 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.534117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:43167 (local address 127.0.0.1:38826)
0926 22:42:50.534474 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:50.534517 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338}
W20260926 22:42:50.536222 27025 catalog_manager.cc:4584] TS 12a5559fe9574cdeaecea34832279b54 (127.11.107.193:43167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3be6188cbbae4da9ab17b37979543c9b: Network error: Client connection negotiation failed: client connection to 127.11.107.193:43167: connect: Connection refused (error 111)
I20260926 22:42:50.539048 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 27371
I20260926 22:42:50.539230 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:42:50.539083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:34173 (local address 127.0.0.1:47488)
0926 22:42:50.539173 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 22:42:50.539196 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.194:34173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 22:42:50.543828 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 27009
I20260926 22:43:10.548981 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:45429
--webserver_interface=127.11.107.254
--webserver_port=46161
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:45429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:10.629331 27543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:10.629593 27543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:10.629632 27543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:10.630807 27543 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 22:43:10.630875 27543 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 22:43:10.630897 27543 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 22:43:10.630916 27543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:43:10.630944 27543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:10.630975 27543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:10.631003 27543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:43:10.631031 27543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:43:10.632516 27543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:45429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:45429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=46161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:10.632812 27543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:10.633042 27543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:43:10.641901 27543 server_base.cc:1034] running on GCE node
W20260926 22:43:10.642144 27549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:10.642144 27552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:10.642204 27550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:10.642525 27543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:10.642870 27543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:10.644057 27543 hybrid_clock.cc:648] HybridClock initialized: now 1790462590644052 us; error 37 us; skew 500 ppm
Sep 26 22:43:10 dist-test-slave-j8sq krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462590, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:10.646771 27543 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:43:10.647266 27543 webserver.cc:460] Webserver started at http://127.11.107.254:46161/ using document root <none> and password file <none>
I20260926 22:43:10.647502 27543 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:10.647586 27543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:10.648962 27543 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:43:10.649634 27559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:10.649811 27543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:10.649952 27543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "5aaee8f0076547e582f3d5bc7cbcf109"
format_stamp: "Formatted at 2026-09-26 22:42:49 on dist-test-slave-j8sq"
server_key: "ae34ba4d6744ba0aca1e89fb02feddee"
server_key_iv: "d488b20c15fd66637896f5ce9bb68aed"
server_key_version: "encryptionkey@0"
I20260926 22:43:10.650359 27543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:10.660471 27543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:10.662719 27543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:10.662884 27543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:10.666886 27543 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:45429
I20260926 22:43:10.666929 27611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:45429 every 8 connection(s)
I20260926 22:43:10.667320 27543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:43:10.668782 27612 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:43:10.669404 27612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Bootstrap starting.
I20260926 22:43:10.673988 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 27543
I20260926 22:43:10.674335 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:43167
--local_ip_for_outbound_sockets=127.11.107.193
--tserver_master_addrs=127.11.107.254:45429
--webserver_port=42561
--webserver_interface=127.11.107.193
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:43:10.678486 27612 log.cc:824] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:10.680538 27612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:43:10.680876 27612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Bootstrap complete.
I20260926 22:43:10.683185 27612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } }
I20260926 22:43:10.683405 27612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aaee8f0076547e582f3d5bc7cbcf109, State: Initialized, Role: FOLLOWER
I20260926 22:43:10.683576 27612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } }
I20260926 22:43:10.683641 27612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:43:10.683697 27612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:43:10.683769 27612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:43:10.684798 27612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } }
I20260926 22:43:10.684911 27612 leader_election.cc:302] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aaee8f0076547e582f3d5bc7cbcf109; no voters:
I20260926 22:43:10.685127 27612 leader_election.cc:288] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 22:43:10.685161 27616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 22:43:10.685276 27616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 2 LEADER]: Becoming Leader. State: Replica: 5aaee8f0076547e582f3d5bc7cbcf109, State: Running, Role: LEADER
I20260926 22:43:10.685389 27612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:43:10.685431 27616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } }
I20260926 22:43:10.685845 27617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } } }
I20260926 22:43:10.685977 27617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:10.686213 27618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aaee8f0076547e582f3d5bc7cbcf109. Latest consensus state: current_term: 2 leader_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } } }
I20260926 22:43:10.686297 27618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:10.686534 27623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:43:10.687321 27623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:43:10.687425 27623 catalog_manager.cc:1259] Loaded cluster ID: f27c5eae9f424fbdb3f97e2dae3d0b42
I20260926 22:43:10.687505 27623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:43:10.688508 27623 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:43:10.688709 27623 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Loaded TSK: 0
I20260926 22:43:10.701032 27623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Generated new TSK 1
I20260926 22:43:10.701148 27623 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 22:43:10.770273 27614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:10.770506 27614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:10.770545 27614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:10.771664 27614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:10.771731 27614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:10.771780 27614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:43:10.773262 27614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:43167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=42561
--tserver_master_addrs=127.11.107.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:10.773530 27614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:10.773777 27614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:10.779716 27642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:10.779903 27640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:10.779919 27639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:10.780260 27614 server_base.cc:1034] running on GCE node
I20260926 22:43:10.780469 27614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:10.780750 27614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:10.781922 27614 hybrid_clock.cc:648] HybridClock initialized: now 1790462590781903 us; error 33 us; skew 500 ppm
Sep 26 22:43:10 dist-test-slave-j8sq krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462590, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:10.784874 27614 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:43:10.785316 27614 webserver.cc:460] Webserver started at http://127.11.107.193:42561/ using document root <none> and password file <none>
I20260926 22:43:10.785578 27614 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:10.785766 27614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:10.787355 27614 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:10.788084 27649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:10.788269 27614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:10.788367 27614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "12a5559fe9574cdeaecea34832279b54"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "e1345e8b75511a1dd6d60e269f62dba8"
server_key_iv: "585ccdb3cb026d9169457b656366c835"
server_key_version: "encryptionkey@0"
I20260926 22:43:10.788710 27614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:10.806900 27614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:10.809342 27614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:10.809487 27614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:10.809780 27614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:10.810294 27656 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 22:43:10.812099 27614 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 22:43:10.812158 27614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 22:43:10.812183 27614 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 22:43:10.812853 27614 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 22:43:10.812911 27614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 22:43:10.812964 27656 tablet_bootstrap.cc:492] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Bootstrap starting.
I20260926 22:43:10.820001 27614 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:43167
I20260926 22:43:10.820358 27614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:43:10.822484 27763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:43167 every 8 connection(s)
I20260926 22:43:10.823607 27656 log.cc:824] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:10.826145 27656 tablet_bootstrap.cc:492] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:43:10.826462 27656 tablet_bootstrap.cc:492] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Bootstrap complete.
I20260926 22:43:10.827647 27656 ts_tablet_manager.cc:1397] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
Sep 26 22:43:10 dist-test-slave-j8sq krb5kdc[26996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462590, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:10.828969 27656 raft_consensus.cc:348] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:10.829317 27656 raft_consensus.cc:729] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a5559fe9574cdeaecea34832279b54, State: Initialized, Role: FOLLOWER
I20260926 22:43:10.829538 27656 consensus_queue.cc:260] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:10.829674 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 27614
I20260926 22:43:10.829854 27656 ts_tablet_manager.cc:1428] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:43:10.829978 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:34173
--local_ip_for_outbound_sockets=127.11.107.194
--tserver_master_addrs=127.11.107.254:45429
--webserver_port=40617
--webserver_interface=127.11.107.194
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:43:10.834280 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:10.821042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:42839)
0926 22:43:10.821239 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:10.821280 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:10.821711 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:10.822359 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:10.822363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:10.822731 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:10.822905 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:10.822910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:10.824188 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:10.824189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:10.825411 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:10.825414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:10.826357 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:10.826359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:10.826504 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:10.827096 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:10.827105 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:10.828838 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:10.832460 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:10.832467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:10.832469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:10.832753 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:10.833054 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:10.833055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:10.833056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:10.833180 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:10.833655 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:10.833662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:10.833810 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 22:43:10.833942 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 22:43:10.834093 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":119,"threads_started":1}
I20260926 22:43:10.834390 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:10.821128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:42839 (local address 127.11.107.254:45429)
0926 22:43:10.822013 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:10.822020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:10.822048 (+ 28us) server_negotiation.cc:407] Connection header received
0926 22:43:10.822148 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:10.822153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:10.822190 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:10.822331 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:10.823019 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:10.824038 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:10.825513 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:10.826137 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:10.826220 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:10.828955 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:10.828959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:10.828959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:10.828970 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:10.832356 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:10.832850 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:10.832853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:10.832854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:10.832969 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:10.833252 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:10.833253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:10.833254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:10.833549 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:10.833651 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:10.834007 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 22:43:10.834157 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":74,"threads_started":1}
I20260926 22:43:10.835395 27764 heartbeater.cc:347] Connected to a master server at 127.11.107.254:45429
I20260926 22:43:10.835492 27764 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:10.835680 27764 heartbeater.cc:511] Master 127.11.107.254:45429 requested a full tablet report, sending...
I20260926 22:43:10.836477 27576 ts_manager.cc:194] Registered new tserver with Master: 12a5559fe9574cdeaecea34832279b54 (127.11.107.193:43167)
I20260926 22:43:10.837723 27576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:42839
I20260926 22:43:10.838601 27764 heartbeater.cc:503] Master 127.11.107.254:45429 was elected leader, sending a full tablet report...
Sep 26 22:43:10 dist-test-slave-j8sq krb5kdc[26996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462590, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for kudu/127.11.107.193@KRBTEST.COM
I20260926 22:43:10.845466 27772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:10.837231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48154 (local address 127.11.107.193:43167)
0926 22:43:10.837388 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:10.837392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:10.837543 (+ 151us) server_negotiation.cc:407] Connection header received
0926 22:43:10.837656 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:10.837658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:10.837681 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:10.837765 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:10.838268 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:10.839459 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:10.840163 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:10.840675 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:10.840750 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:10.842843 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:10.842850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:10.842850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:10.842856 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:10.844365 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:10.844767 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:10.844769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:10.844770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:10.844818 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:10.844999 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:10.844999 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:10.844999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:10.845110 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:10.845182 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:10.845341 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 22:43:10.845409 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260926 22:43:10.845516 27771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:10.837124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:43167 (local address 127.0.0.1:48154)
0926 22:43:10.837439 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:10.837474 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:10.837563 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:10.837768 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:10.837773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:10.838131 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:10.838196 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:10.838198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:10.839540 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:10.839541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:10.840079 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:10.840081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:10.840762 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:10.840762 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:10.840813 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:10.841259 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:10.841264 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:10.842766 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:10.844437 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:10.844443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:10.844445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:10.844692 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:10.844880 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:10.844881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:10.844881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:10.844943 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:10.845167 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:10.845173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:10.845227 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 22:43:10.845313 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 22:43:10.845421 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":89,"threads_started":1}
I20260926 22:43:10.846218 27698 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be6188cbbae4da9ab17b37979543c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:42:50 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:48154
I20260926 22:43:10.846791 27773 tablet_replica.cc:330] stopping tablet replica
I20260926 22:43:10.846874 27773 raft_consensus.cc:2229] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:43:10.846956 27773 raft_consensus.cc:2258] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:10.847224 27773 ts_tablet_manager.cc:1905] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:10.849273 27773 ts_tablet_manager.cc:1918] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:10.849332 27773 log.cc:1196] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal/wals/3be6188cbbae4da9ab17b37979543c9b
I20260926 22:43:10.849587 27773 ts_tablet_manager.cc:1939] T 3be6188cbbae4da9ab17b37979543c9b P 12a5559fe9574cdeaecea34832279b54: Deleting consensus metadata
I20260926 22:43:10.850126 27560 catalog_manager.cc:4841] TS 12a5559fe9574cdeaecea34832279b54 (127.11.107.193:43167): tablet 3be6188cbbae4da9ab17b37979543c9b (table test-table [id=53563fe41884460ba345ef4478789143]) successfully deleted
W20260926 22:43:10.913185 27770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:10.913398 27770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:10.913445 27770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:10.914816 27770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:10.914876 27770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:10.914920 27770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:43:10.916324 27770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:34173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=40617
--tserver_master_addrs=127.11.107.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:10.916569 27770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:10.916795 27770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:10.921497 27778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:10.921470 27781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:10.921469 27779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:10.921718 27770 server_base.cc:1034] running on GCE node
I20260926 22:43:10.921957 27770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:10.922158 27770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:10.923321 27770 hybrid_clock.cc:648] HybridClock initialized: now 1790462590923308 us; error 27 us; skew 500 ppm
Sep 26 22:43:10 dist-test-slave-j8sq krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462590, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:10.925697 27770 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:43:10.926155 27770 webserver.cc:460] Webserver started at http://127.11.107.194:40617/ using document root <none> and password file <none>
I20260926 22:43:10.926329 27770 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:10.926378 27770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:10.927592 27770 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:43:10.928115 27788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:10.928280 27770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:43:10.928375 27770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "477ee41f35fa4b91a2361be5bbeb098b"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "ecf0530fd07eee960b04b19a01004e53"
server_key_iv: "5316a1c1c2d77f648c13189f5ae30d93"
server_key_version: "encryptionkey@0"
I20260926 22:43:10.928694 27770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:10.946719 27770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:10.948418 27770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:10.948580 27770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:10.948797 27770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:10.949112 27770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:10.949167 27770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:10.949201 27770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:10.949229 27770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:10.955024 27770 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:34173
I20260926 22:43:10.955090 27901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:34173 every 8 connection(s)
I20260926 22:43:10.955418 27770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:43:10 dist-test-slave-j8sq krb5kdc[26996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462590, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:10.963999 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:10.956058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:55523 (local address 127.11.107.254:45429)
0926 22:43:10.956107 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:10.956110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:10.956321 (+ 211us) server_negotiation.cc:407] Connection header received
0926 22:43:10.956776 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:10.956776 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:10.956788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:10.956848 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:10.957709 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:10.958277 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:10.959198 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:10.959384 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:10.959435 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:10.961456 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:10.961458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:10.961458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:10.961465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:10.962689 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:10.963083 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:10.963084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:10.963085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:10.963144 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:10.963387 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:10.963388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:10.963388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:10.963494 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:10.963574 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:10.963829 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 22:43:10.963911 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:43:10.964181 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:10.956034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:55523)
0926 22:43:10.956197 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:10.956237 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:10.956696 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:10.956901 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:10.956904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:10.957463 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:10.957635 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:10.957639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:10.958354 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:10.958355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:10.959123 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:10.959125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:10.959451 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:10.959452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:10.959548 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:10.960041 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:10.960049 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:10.961361 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:10.962784 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:10.962790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:10.962792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:10.963000 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:10.963211 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:10.963211 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:10.963212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:10.963312 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:10.963560 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:10.963566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:10.963698 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 22:43:10.963807 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 22:43:10.963951 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260926 22:43:10.964653 27902 heartbeater.cc:347] Connected to a master server at 127.11.107.254:45429
I20260926 22:43:10.964742 27902 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:10.964869 27902 heartbeater.cc:511] Master 127.11.107.254:45429 requested a full tablet report, sending...
I20260926 22:43:10.965250 27576 ts_manager.cc:194] Registered new tserver with Master: 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173)
I20260926 22:43:10.965255 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 27770
I20260926 22:43:10.965518 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:33841
--local_ip_for_outbound_sockets=127.11.107.195
--tserver_master_addrs=127.11.107.254:45429
--webserver_port=44915
--webserver_interface=127.11.107.195
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:43:10.965619 27576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:55523
W20260926 22:43:11.046083 27905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:11.046278 27905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:11.046301 27905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:11.047405 27905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:11.047441 27905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:11.047466 27905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:43:11.048890 27905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:33841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=44915
--tserver_master_addrs=127.11.107.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:11.049077 27905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:11.049257 27905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:11.054270 27911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:11.054260 27910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:11.054318 27905 server_base.cc:1034] running on GCE node
W20260926 22:43:11.054260 27913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:11.054637 27905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:11.054884 27905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:11.056022 27905 hybrid_clock.cc:648] HybridClock initialized: now 1790462591056012 us; error 33 us; skew 500 ppm
Sep 26 22:43:11 dist-test-slave-j8sq krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462591, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:11.058432 27905 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:43:11.058866 27905 webserver.cc:460] Webserver started at http://127.11.107.195:44915/ using document root <none> and password file <none>
I20260926 22:43:11.059060 27905 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:11.059118 27905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:11.060468 27905 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:11.061043 27920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:11.061250 27905 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 22:43:11.061335 27905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "91f3775fd6e747dc806b4fe0b8e8ec72"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "abd7553aaf70ef6b544e55b8389d91f6"
server_key_iv: "e294e061091d1ef94fe5b3f48ad40d4b"
server_key_version: "encryptionkey@0"
I20260926 22:43:11.061638 27905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:11.092334 27905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:11.094288 27905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:11.094393 27905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:11.094573 27905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:11.094866 27905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:11.094901 27905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:11.094924 27905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:11.094936 27905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:11.100716 27905 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:33841
I20260926 22:43:11.100812 28033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:33841 every 8 connection(s)
I20260926 22:43:11.101126 27905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:43:11 dist-test-slave-j8sq krb5kdc[26996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462591, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:11.110191 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.101943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:34531 (local address 127.11.107.254:45429)
0926 22:43:11.102014 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:11.102016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:11.102217 (+ 201us) server_negotiation.cc:407] Connection header received
0926 22:43:11.102714 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:11.102716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:11.102726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:11.102787 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:11.103691 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.104193 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.105132 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.105309 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.105361 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.107501 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:11.107502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:11.107503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:11.107511 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:11.108739 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:11.109176 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:11.109178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:11.109178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:11.109240 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:11.109487 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:11.109488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:11.109488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:11.109605 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:11.109682 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:11.110009 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 22:43:11.110096 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 22:43:11.110365 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.101883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:34531)
0926 22:43:11.102088 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:11.102125 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:11.102588 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:11.102864 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:11.102868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:11.103416 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:11.103596 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:11.103601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.104270 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.104271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:11.105050 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:11.105052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.105400 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.105401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:11.105520 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.106097 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:11.106105 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:11.107400 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:11.108832 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:11.108838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:11.108840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:11.109070 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:11.109309 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:11.109310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:11.109310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:11.109415 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:11.109674 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:11.109680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:11.109829 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 22:43:11.109975 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 22:43:11.110124 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":107,"threads_started":1}
I20260926 22:43:11.110710 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 27905
I20260926 22:43:11.110884 28034 heartbeater.cc:347] Connected to a master server at 127.11.107.254:45429
I20260926 22:43:11.110980 28034 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:11.111163 28034 heartbeater.cc:511] Master 127.11.107.254:45429 requested a full tablet report, sending...
I20260926 22:43:11.111567 27576 ts_manager.cc:194] Registered new tserver with Master: 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841)
I20260926 22:43:11.112097 27576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:34531
I20260926 22:43:11.116998 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.111113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.11.107.254:45429)
0926 22:43:11.111164 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:11.111166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:11.111370 (+ 204us) server_negotiation.cc:407] Connection header received
0926 22:43:11.111455 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:11.111456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:11.111466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:11.111525 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:11.112264 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.112731 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.113594 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.113776 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.113834 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.115044 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:11.115045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:11.115045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:11.115051 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:11.116151 (+ 1100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:11.116450 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:11.116451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:11.116451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:11.116496 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:11.116630 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:11.116630 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:11.116631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:11.116718 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:11.116830 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:11.116893 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 22:43:11.116944 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:43:11.117741 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:43:11.120174 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.118093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37018 (local address 127.11.107.254:45429)
0926 22:43:11.118140 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:11.118142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:11.118280 (+ 138us) server_negotiation.cc:407] Connection header received
0926 22:43:11.118346 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:11.118346 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:11.118371 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:11.118425 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:43:11.118502 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.118920 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.119603 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.119782 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.119842 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.119962 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:43:11.119996 (+ 34us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 22:43:11.120104 (+ 108us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37018: token expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 22:43:11.120226 27767 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37018: token expired
W20260926 22:43:11.120342 27506 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.107.254:45429)
I20260926 22:43:11.125941 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.120554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.11.107.254:45429)
0926 22:43:11.120620 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:11.120622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:11.120642 (+ 20us) server_negotiation.cc:407] Connection header received
0926 22:43:11.120699 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:11.120700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:11.120706 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:11.120749 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:11.121159 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.121580 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.122288 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.122451 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.122504 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.123585 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:11.123587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:11.123587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:11.123594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:11.124815 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:11.125115 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:11.125116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:11.125116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:11.125163 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:11.125350 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:11.125351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:11.125351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:11.125458 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:11.125535 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:11.125674 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 22:43:11.125786 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 22:43:11.127023 27576 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:43:11.128583 27576 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:43:11.130755 27698 tablet_service.cc:1467] Processing CreateTablet for tablet b0f60f20f15342638c0edcfe2c759b05 (DEFAULT_TABLE table=test-table [id=cbaf997efe934871aa81abdfcd54b6b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:11.130954 27698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f60f20f15342638c0edcfe2c759b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:11.132471 27656 tablet_bootstrap.cc:492] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54: Bootstrap starting.
I20260926 22:43:11.132767 27656 tablet_bootstrap.cc:654] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:11.133383 27656 tablet_bootstrap.cc:492] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54: No bootstrap required, opened a new log
I20260926 22:43:11.133433 27656 ts_tablet_manager.cc:1397] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:11.133812 27656 raft_consensus.cc:348] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:11.133913 27656 raft_consensus.cc:374] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:11.133963 27656 raft_consensus.cc:729] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a5559fe9574cdeaecea34832279b54, State: Initialized, Role: FOLLOWER
I20260926 22:43:11.134004 27656 consensus_queue.cc:260] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:11.134085 27656 ts_tablet_manager.cc:1428] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:11.134327 28039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.130373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:34173 (local address 127.0.0.1:53762)
0926 22:43:11.130488 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:11.130499 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:11.130549 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:11.130888 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:11.130889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:11.130892 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:11.130936 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:11.130937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.132098 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.132099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:11.133105 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:11.133108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.134061 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.134062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:11.134127 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.134147 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 22:43:11.134198 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:43:11.134250 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":11,"thread_start_us":56,"threads_started":1}
I20260926 22:43:11.134567 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.130478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.11.107.194:34173)
0926 22:43:11.130696 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:11.130701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:11.130711 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:43:11.130752 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:11.130754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:11.130769 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:11.130856 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:11.130998 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.132015 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.133208 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.133906 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.133974 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.134028 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:11.134425 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 22:43:11.134478 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":122,"threads_started":1}
I20260926 22:43:11.135226 27836 tablet_service.cc:1467] Processing CreateTablet for tablet b0f60f20f15342638c0edcfe2c759b05 (DEFAULT_TABLE table=test-table [id=cbaf997efe934871aa81abdfcd54b6b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:11.135526 27836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f60f20f15342638c0edcfe2c759b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:11.136701 27771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.130373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:33841 (local address 127.0.0.1:58490)
0926 22:43:11.130472 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:11.130484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:11.130524 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:11.133399 (+ 2875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:11.133400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:11.133402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:11.133445 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:11.133447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.134781 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.134782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:11.135632 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:11.135635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.136513 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.136514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:11.136559 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.136563 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:43:11.136604 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 22:43:11.136642 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 22:43:11.136749 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.133026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58490 (local address 127.11.107.195:33841)
0926 22:43:11.133190 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:11.133195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:11.133229 (+ 34us) server_negotiation.cc:407] Connection header received
0926 22:43:11.133270 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:11.133272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:11.133288 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:11.133529 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:11.133615 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.134706 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.135711 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.136429 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.136510 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.136559 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:11.136634 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 22:43:11.136671 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260926 22:43:11.138413 27968 tablet_service.cc:1467] Processing CreateTablet for tablet b0f60f20f15342638c0edcfe2c759b05 (DEFAULT_TABLE table=test-table [id=cbaf997efe934871aa81abdfcd54b6b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:11.138773 27968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f60f20f15342638c0edcfe2c759b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:11.138813 28043 tablet_bootstrap.cc:492] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b: Bootstrap starting.
I20260926 22:43:11.139441 28043 tablet_bootstrap.cc:654] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:11.140021 28043 log.cc:824] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:11.140822 28043 tablet_bootstrap.cc:492] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b: No bootstrap required, opened a new log
I20260926 22:43:11.140878 28043 ts_tablet_manager.cc:1397] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:11.142450 28045 tablet_bootstrap.cc:492] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72: Bootstrap starting.
I20260926 22:43:11.142876 28043 raft_consensus.cc:348] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:11.143016 28043 raft_consensus.cc:374] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:11.143039 28043 raft_consensus.cc:729] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477ee41f35fa4b91a2361be5bbeb098b, State: Initialized, Role: FOLLOWER
I20260926 22:43:11.143057 28045 tablet_bootstrap.cc:654] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:11.143196 28043 consensus_queue.cc:260] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:11.143466 28043 ts_tablet_manager.cc:1428] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:43:11.143530 27902 heartbeater.cc:503] Master 127.11.107.254:45429 was elected leader, sending a full tablet report...
I20260926 22:43:11.143714 28045 log.cc:824] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:11.144649 28045 tablet_bootstrap.cc:492] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72: No bootstrap required, opened a new log
I20260926 22:43:11.144726 28045 ts_tablet_manager.cc:1397] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:11.146512 28045 raft_consensus.cc:348] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:11.146709 28045 raft_consensus.cc:374] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:11.146756 28045 raft_consensus.cc:729] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f3775fd6e747dc806b4fe0b8e8ec72, State: Initialized, Role: FOLLOWER
I20260926 22:43:11.146878 28045 consensus_queue.cc:260] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:11.147156 28045 ts_tablet_manager.cc:1428] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:43:11.147173 28034 heartbeater.cc:503] Master 127.11.107.254:45429 was elected leader, sending a full tablet report...
I20260926 22:43:11.202704 27769 raft_consensus.cc:482] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:11.202880 27769 raft_consensus.cc:504] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:11.203267 27769 leader_election.cc:288] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173), 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841)
W20260926 22:43:11.205992 27903 tablet.cc:2369] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:43:11.206352 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.203511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:58445 (local address 127.11.107.194:34173)
0926 22:43:11.203557 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:11.203560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:11.203567 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:43:11.203596 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:11.203596 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:11.203606 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:11.203667 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:11.203878 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.204442 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.205516 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.205954 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.206028 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.206038 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:11.206241 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 22:43:11.206281 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 22:43:11.206359 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.203393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:34173 (local address 127.11.107.193:58445)
0926 22:43:11.203429 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:11.203443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:11.203499 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:11.203710 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:11.203712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:11.203714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:11.203797 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:11.203800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.204525 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.204527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:11.205392 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:11.205394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.206046 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.206047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:11.206125 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.206151 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 22:43:11.206223 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 22:43:11.206271 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":31}
I20260926 22:43:11.206461 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.203520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:44723 (local address 127.11.107.195:33841)
0926 22:43:11.203569 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:11.203572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:11.203695 (+ 123us) server_negotiation.cc:407] Connection header received
0926 22:43:11.203775 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:11.203776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:11.203787 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:11.203851 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:11.204027 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.204498 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.205488 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.206014 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.206095 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.206104 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:11.206324 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 22:43:11.206368 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:43:11.206691 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.203430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:33841 (local address 127.11.107.193:44723)
0926 22:43:11.203620 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:11.203629 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:11.203704 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:11.203876 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:11.203877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:11.203879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:11.203957 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:11.203959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.204586 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.204587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:11.205392 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:11.205394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.206149 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.206150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:11.206233 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.206236 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:43:11.206535 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 22:43:11.206590 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":2,"thread_start_us":142,"threads_started":1}
I20260926 22:43:11.206722 27856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f60f20f15342638c0edcfe2c759b05" candidate_uuid: "12a5559fe9574cdeaecea34832279b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477ee41f35fa4b91a2361be5bbeb098b" is_pre_election: true
I20260926 22:43:11.206878 27856 raft_consensus.cc:2454] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12a5559fe9574cdeaecea34832279b54 in term 0.
I20260926 22:43:11.207139 27651 leader_election.cc:302] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a5559fe9574cdeaecea34832279b54, 477ee41f35fa4b91a2361be5bbeb098b; no voters:
I20260926 22:43:11.207233 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f60f20f15342638c0edcfe2c759b05" candidate_uuid: "12a5559fe9574cdeaecea34832279b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" is_pre_election: true
I20260926 22:43:11.207299 27769 raft_consensus.cc:2781] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:43:11.207376 27769 raft_consensus.cc:482] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:43:11.207376 27988 raft_consensus.cc:2454] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12a5559fe9574cdeaecea34832279b54 in term 0.
I20260926 22:43:11.207414 27769 raft_consensus.cc:3037] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:11.208210 27769 raft_consensus.cc:504] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:11.208366 27769 leader_election.cc:288] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 1 election: Requested vote from peers 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173), 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841)
I20260926 22:43:11.208510 27856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f60f20f15342638c0edcfe2c759b05" candidate_uuid: "12a5559fe9574cdeaecea34832279b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477ee41f35fa4b91a2361be5bbeb098b"
I20260926 22:43:11.208515 27988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f60f20f15342638c0edcfe2c759b05" candidate_uuid: "12a5559fe9574cdeaecea34832279b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72"
I20260926 22:43:11.208600 27856 raft_consensus.cc:3037] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:11.208600 27988 raft_consensus.cc:3037] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:11.209510 27856 raft_consensus.cc:2454] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12a5559fe9574cdeaecea34832279b54 in term 1.
I20260926 22:43:11.209510 27988 raft_consensus.cc:2454] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12a5559fe9574cdeaecea34832279b54 in term 1.
I20260926 22:43:11.209695 27651 leader_election.cc:302] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a5559fe9574cdeaecea34832279b54, 477ee41f35fa4b91a2361be5bbeb098b; no voters:
I20260926 22:43:11.209813 27769 raft_consensus.cc:2781] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:11.209976 27769 raft_consensus.cc:686] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 LEADER]: Becoming Leader. State: Replica: 12a5559fe9574cdeaecea34832279b54, State: Running, Role: LEADER
I20260926 22:43:11.210070 27769 consensus_queue.cc:237] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:11.210855 27576 catalog_manager.cc:5495] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12a5559fe9574cdeaecea34832279b54 (127.11.107.193). New cstate: current_term: 1 leader_uuid: "12a5559fe9574cdeaecea34832279b54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } health_report { overall_health: HEALTHY } } }
I20260926 22:43:11.225644 27772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.223537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.11.107.193:43167)
0926 22:43:11.223598 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:11.223600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:11.223607 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:43:11.223673 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:11.223673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:11.223689 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:11.223743 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:43:11.223856 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.224319 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.224992 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:11.225181 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:11.225229 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:11.225349 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:43:11.225435 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:43:11.225487 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:11.225528 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 22:43:11.225544 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 22:43:11.227075 27856 raft_consensus.cc:1261] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b [term 1 FOLLOWER]: Refusing update from remote peer 12a5559fe9574cdeaecea34832279b54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:11.227075 27988 raft_consensus.cc:1261] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 1 FOLLOWER]: Refusing update from remote peer 12a5559fe9574cdeaecea34832279b54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:11.227411 27769 consensus_queue.cc:1035] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:11.227560 28050 consensus_queue.cc:1035] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:11.229781 28053 mvcc.cc:205] Tried to move back new op lower bound from 7333734773663567872 to 7333734773597126656. Current Snapshot: MvccSnapshot[applied={T|T < 7333734773663567872}]
I20260926 22:43:11.229826 28055 mvcc.cc:205] Tried to move back new op lower bound from 7333734773663567872 to 7333734773597126656. Current Snapshot: MvccSnapshot[applied={T|T < 7333734773663567872}]
I20260926 22:43:11.230173 28054 mvcc.cc:205] Tried to move back new op lower bound from 7333734773663567872 to 7333734773597126656. Current Snapshot: MvccSnapshot[applied={T|T < 7333734773663567872}]
I20260926 22:43:11.232214 27576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:11.232314 27576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:11.232973 27576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Sending DeleteTablet for 3 replicas of tablet b0f60f20f15342638c0edcfe2c759b05
I20260926 22:43:11.233311 27836 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f60f20f15342638c0edcfe2c759b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:11 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:53762
I20260926 22:43:11.233330 27968 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f60f20f15342638c0edcfe2c759b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:11 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:58490
I20260926 22:43:11.233547 28063 tablet_replica.cc:330] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72: stopping tablet replica
I20260926 22:43:11.233558 27698 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f60f20f15342638c0edcfe2c759b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:11 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:48154
I20260926 22:43:11.233566 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 27614
I20260926 22:43:11.233629 27773 tablet_replica.cc:330] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54: stopping tablet replica
I20260926 22:43:11.233680 28063 raft_consensus.cc:2229] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:43:11.233691 27773 raft_consensus.cc:2229] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:43:11.233769 27773 raft_consensus.cc:2258] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:11.233777 28063 raft_consensus.cc:2258] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:11.233932 28064 tablet_replica.cc:330] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b: stopping tablet replica
I20260926 22:43:11.234005 28064 raft_consensus.cc:2229] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:43:11.234084 28064 raft_consensus.cc:2258] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:11.234571 28063 ts_tablet_manager.cc:1905] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:11.234616 28064 ts_tablet_manager.cc:1905] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:11.236703 28063 ts_tablet_manager.cc:1918] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:11.236783 28063 log.cc:1196] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/b0f60f20f15342638c0edcfe2c759b05
I20260926 22:43:11.236747 28064 ts_tablet_manager.cc:1918] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:11.236886 28064 log.cc:1196] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/b0f60f20f15342638c0edcfe2c759b05
I20260926 22:43:11.237068 28063 ts_tablet_manager.cc:1939] T b0f60f20f15342638c0edcfe2c759b05 P 91f3775fd6e747dc806b4fe0b8e8ec72: Deleting consensus metadata
I20260926 22:43:11.237253 28064 ts_tablet_manager.cc:1939] T b0f60f20f15342638c0edcfe2c759b05 P 477ee41f35fa4b91a2361be5bbeb098b: Deleting consensus metadata
I20260926 22:43:11.237473 27562 catalog_manager.cc:4841] TS 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841): tablet b0f60f20f15342638c0edcfe2c759b05 (table test-table [id=cbaf997efe934871aa81abdfcd54b6b5]) successfully deleted
I20260926 22:43:11.237645 27561 catalog_manager.cc:4841] TS 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173): tablet b0f60f20f15342638c0edcfe2c759b05 (table test-table [id=cbaf997efe934871aa81abdfcd54b6b5]) successfully deleted
W20260926 22:43:11.239555 27560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:43167 (error 108)
I20260926 22:43:11.239836 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 27770
I20260926 22:43:11.240394 27771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:11.239929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:43167 (local address 127.0.0.1:48184)
0926 22:43:11.240319 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:11.240357 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371}
W20260926 22:43:11.241218 27560 catalog_manager.cc:4584] TS 12a5559fe9574cdeaecea34832279b54 (127.11.107.193:43167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0f60f20f15342638c0edcfe2c759b05: Network error: Client connection negotiation failed: client connection to 127.11.107.193:43167: connect: Connection refused (error 111)
I20260926 22:43:11.244966 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 27905
I20260926 22:43:11.249465 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 27543
I20260926 22:43:31.254796 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:45429
--webserver_interface=127.11.107.254
--webserver_port=46161
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:45429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:31.335211 28066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:31.335412 28066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:31.335436 28066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:31.336619 28066 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 22:43:31.336663 28066 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 22:43:31.336678 28066 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 22:43:31.336691 28066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:43:31.336747 28066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:31.336768 28066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:31.336789 28066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:43:31.336824 28066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:43:31.338382 28066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:45429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:45429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=46161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:31.338639 28066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:31.338879 28066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:31.343966 28071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:31.343967 28072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:31.344017 28066 server_base.cc:1034] running on GCE node
W20260926 22:43:31.343957 28074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:31.344444 28066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:31.344744 28066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:31.345911 28066 hybrid_clock.cc:648] HybridClock initialized: now 1790462611345890 us; error 26 us; skew 500 ppm
Sep 26 22:43:31 dist-test-slave-j8sq krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:31.348338 28066 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:43:31.348739 28066 webserver.cc:460] Webserver started at http://127.11.107.254:46161/ using document root <none> and password file <none>
I20260926 22:43:31.348966 28066 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:31.349035 28066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:31.350409 28066 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:43:31.351056 28081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:31.351227 28066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:43:31.351323 28066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "5aaee8f0076547e582f3d5bc7cbcf109"
format_stamp: "Formatted at 2026-09-26 22:42:49 on dist-test-slave-j8sq"
server_key: "ae34ba4d6744ba0aca1e89fb02feddee"
server_key_iv: "d488b20c15fd66637896f5ce9bb68aed"
server_key_version: "encryptionkey@0"
I20260926 22:43:31.351639 28066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:31.362807 28066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:31.364615 28066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:31.364763 28066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:31.368955 28066 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:45429
I20260926 22:43:31.368985 28133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:45429 every 8 connection(s)
I20260926 22:43:31.369376 28066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:43:31.370783 28134 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:43:31.371351 28134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Bootstrap starting.
I20260926 22:43:31.379422 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 28066
I20260926 22:43:31.379544 28134 log.cc:824] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:31.379746 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:43167
--local_ip_for_outbound_sockets=127.11.107.193
--tserver_master_addrs=127.11.107.254:45429
--webserver_port=42561
--webserver_interface=127.11.107.193
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:43:31.382335 28134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:43:31.382638 28134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Bootstrap complete.
I20260926 22:43:31.386536 28134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } }
I20260926 22:43:31.386930 28134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aaee8f0076547e582f3d5bc7cbcf109, State: Initialized, Role: FOLLOWER
I20260926 22:43:31.387125 28134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } }
I20260926 22:43:31.387213 28134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:43:31.387261 28134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:43:31.387316 28134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 2 FOLLOWER]: Advancing to term 3
I20260926 22:43:31.388873 28134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } }
I20260926 22:43:31.388993 28134 leader_election.cc:302] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aaee8f0076547e582f3d5bc7cbcf109; no voters:
I20260926 22:43:31.389166 28134 leader_election.cc:288] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 22:43:31.389233 28138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 22:43:31.389393 28138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [term 3 LEADER]: Becoming Leader. State: Replica: 5aaee8f0076547e582f3d5bc7cbcf109, State: Running, Role: LEADER
I20260926 22:43:31.389542 28138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } }
I20260926 22:43:31.389567 28134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:43:31.389882 28139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } } }
I20260926 22:43:31.389953 28139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:31.390309 28142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:43:31.391043 28142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:43:31.391057 28140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aaee8f0076547e582f3d5bc7cbcf109. Latest consensus state: current_term: 3 leader_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaee8f0076547e582f3d5bc7cbcf109" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45429 } } }
I20260926 22:43:31.391498 28140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:31.391245 28142 catalog_manager.cc:1259] Loaded cluster ID: f27c5eae9f424fbdb3f97e2dae3d0b42
I20260926 22:43:31.391559 28142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:43:31.392699 28142 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:43:31.392881 28142 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Loaded TSK: 0
I20260926 22:43:31.392930 28142 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5aaee8f0076547e582f3d5bc7cbcf109: Loaded TSK: 1
F20260926 22:43:31.393141 28142 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2db2fef9ad google::LogMessage::Fail() at ??:0
@ 0x7f2db2ff288a google::LogMessage::SendToLog() at ??:0
@ 0x7f2db2fef4de google::LogMessage::Flush() at ??:0
@ 0x7f2db2ff0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2db575c104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2db5759d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2dbaf491cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2dbaf4948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2dbaf49594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f2dbaf0fc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f2dbaf19ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2db4ebd0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2db4eb2954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2db5d166db start_thread at ??:0
@ 0x7f2db1aed61f clone at ??:0
W20260926 22:43:31.474299 28137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:31.474505 28137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:31.474541 28137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:31.475673 28137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:31.475736 28137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:31.475781 28137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:43:31.477197 28137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:43167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=42561
--tserver_master_addrs=127.11.107.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:31.477453 28137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:31.477681 28137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:31.482520 28178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:31.482805 28175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:31.482806 28176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:31.482802 28137 server_base.cc:1034] running on GCE node
I20260926 22:43:31.483137 28137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:31.483405 28137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:31.484659 28137 hybrid_clock.cc:648] HybridClock initialized: now 1790462611484645 us; error 31 us; skew 500 ppm
Sep 26 22:43:31 dist-test-slave-j8sq krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:31.487226 28137 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:43:31.487717 28137 webserver.cc:460] Webserver started at http://127.11.107.193:42561/ using document root <none> and password file <none>
I20260926 22:43:31.487957 28137 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:31.488003 28137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:31.489306 28137 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:31.490079 28185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:31.490223 28137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:43:31.490324 28137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "12a5559fe9574cdeaecea34832279b54"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "e1345e8b75511a1dd6d60e269f62dba8"
server_key_iv: "585ccdb3cb026d9169457b656366c835"
server_key_version: "encryptionkey@0"
I20260926 22:43:31.490643 28137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:31.504546 28137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:31.506750 28137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:31.506888 28137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:31.507130 28137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:31.507681 28192 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 22:43:31.509578 28137 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 22:43:31.509639 28137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 22:43:31.509686 28137 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 22:43:31.510352 28137 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 22:43:31.510412 28137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 22:43:31.510478 28192 tablet_bootstrap.cc:492] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54: Bootstrap starting.
I20260926 22:43:31.517604 28137 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:43167
I20260926 22:43:31.517650 28299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:43167 every 8 connection(s)
I20260926 22:43:31.518187 28137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:43:31.521381 28192 log.cc:824] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:31.524040 28192 tablet_bootstrap.cc:492] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:43:31.524389 28192 tablet_bootstrap.cc:492] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54: Bootstrap complete.
I20260926 22:43:31.525355 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 28137
I20260926 22:43:31.525617 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:34173
--local_ip_for_outbound_sockets=127.11.107.194
--tserver_master_addrs=127.11.107.254:45429
--webserver_port=40617
--webserver_interface=127.11.107.194
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:43:31.525658 28192 ts_tablet_manager.cc:1397] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260926 22:43:31.527096 28192 raft_consensus.cc:348] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:31.527436 28192 raft_consensus.cc:729] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a5559fe9574cdeaecea34832279b54, State: Initialized, Role: FOLLOWER
I20260926 22:43:31.527693 28192 consensus_queue.cc:260] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:31.527982 28192 ts_tablet_manager.cc:1428] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54: Time spent starting tablet: real 0.002s user 0.004s sys 0.002s
W20260926 22:43:31.600531 28302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:43:31.518821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:38277)
0926 22:43:31.518994 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:31.519038 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:31.519483 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:31.600308 (+ 80825us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
W20260926 22:43:31.600824 28300 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:45429 (0 consecutive failures): Network error: Failed to ping master at 127.11.107.254:45429: Client connection negotiation failed: client connection to 127.11.107.254:45429: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 22:43:31.601154 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:31.601033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:42205)
0926 22:43:31.601087 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:31.601125 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 22:43:31.601538 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:31.601452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:59089)
0926 22:43:31.601501 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:31.601524 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 22:43:31.601694 28300 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 22:43:31.618952 28304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:31.619153 28304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:31.619191 28304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:31.620306 28304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:31.620366 28304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:31.620411 28304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:43:31.621933 28304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:34173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=40617
--tserver_master_addrs=127.11.107.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:31.622164 28304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:31.622388 28304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:31.627111 28310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:31.627084 28313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:31.627113 28311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:31.627410 28304 server_base.cc:1034] running on GCE node
I20260926 22:43:31.627600 28304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:31.627849 28304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:31.629005 28304 hybrid_clock.cc:648] HybridClock initialized: now 1790462611628994 us; error 54 us; skew 500 ppm
Sep 26 22:43:31 dist-test-slave-j8sq krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:31.631554 28304 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:43:31.631951 28304 webserver.cc:460] Webserver started at http://127.11.107.194:40617/ using document root <none> and password file <none>
I20260926 22:43:31.632164 28304 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:31.632236 28304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:31.633558 28304 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:31.634196 28320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:31.634361 28304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:31.634454 28304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "477ee41f35fa4b91a2361be5bbeb098b"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "ecf0530fd07eee960b04b19a01004e53"
server_key_iv: "5316a1c1c2d77f648c13189f5ae30d93"
server_key_version: "encryptionkey@0"
I20260926 22:43:31.634779 28304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:31.645509 28304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:31.647490 28304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:31.647644 28304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:31.647867 28304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:31.648160 28304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:31.648218 28304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:31.648257 28304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:31.648286 28304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:31.654078 28304 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:34173
I20260926 22:43:31.654137 28433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:34173 every 8 connection(s)
I20260926 22:43:31.654490 28304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:43:31.655645 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:31.655141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:37713)
0926 22:43:31.655341 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:31.655432 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
W20260926 22:43:31.656056 28434 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:45429 (0 consecutive failures): Network error: Failed to ping master at 127.11.107.254:45429: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
I20260926 22:43:31.656349 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:31.656245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:42353)
0926 22:43:31.656301 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:31.656325 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 22:43:31.656661 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:31.656547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:34823)
0926 22:43:31.656588 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:31.656610 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 22:43:31.656759 28434 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 22:43:31.661381 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 28304
I20260926 22:43:31.661650 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:33841
--local_ip_for_outbound_sockets=127.11.107.195
--tserver_master_addrs=127.11.107.254:45429
--webserver_port=44915
--webserver_interface=127.11.107.195
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:31.744771 28437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:31.744989 28437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:31.745028 28437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:31.746168 28437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:31.746236 28437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:31.746282 28437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:43:31.747700 28437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:33841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=44915
--tserver_master_addrs=127.11.107.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:31.747964 28437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:31.748189 28437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:31.753046 28443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:31.753141 28437 server_base.cc:1034] running on GCE node
W20260926 22:43:31.753042 28442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:31.753048 28445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:31.753455 28437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:31.753739 28437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:31.754901 28437 hybrid_clock.cc:648] HybridClock initialized: now 1790462611754879 us; error 50 us; skew 500 ppm
Sep 26 22:43:31 dist-test-slave-j8sq krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:31.757493 28437 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:43:31.757982 28437 webserver.cc:460] Webserver started at http://127.11.107.195:44915/ using document root <none> and password file <none>
I20260926 22:43:31.758219 28437 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:31.758286 28437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:31.759559 28437 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:31.760157 28452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:31.760322 28437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:43:31.760424 28437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "91f3775fd6e747dc806b4fe0b8e8ec72"
format_stamp: "Formatted at 2026-09-26 22:42:50 on dist-test-slave-j8sq"
server_key: "abd7553aaf70ef6b544e55b8389d91f6"
server_key_iv: "e294e061091d1ef94fe5b3f48ad40d4b"
server_key_version: "encryptionkey@0"
I20260926 22:43:31.760785 28437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:31.777354 28437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:31.779681 28437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:31.779846 28437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:31.780066 28437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:31.780368 28437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:31.780421 28437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:31.780455 28437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:31.780481 28437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:31.786092 28437 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:33841
I20260926 22:43:31.786274 28565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:33841 every 8 connection(s)
I20260926 22:43:31.786513 28437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:43:31.787469 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:31.787106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:47067)
0926 22:43:31.787266 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:31.787335 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
W20260926 22:43:31.787843 28566 heartbeater.cc:644] Failed to heartbeat to 127.11.107.254:45429 (0 consecutive failures): Network error: Failed to ping master at 127.11.107.254:45429: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
I20260926 22:43:31.788105 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:31.788013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:37801)
0926 22:43:31.788059 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:31.788087 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 22:43:31.788367 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:31.788284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:37371)
0926 22:43:31.788324 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:31.788345 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 22:43:31.788460 28566 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 22:43:31.796392 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 28437
I20260926 22:43:32.602348 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:32.602123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:55409)
0926 22:43:32.602260 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:32.602285 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260926 22:43:32.657296 28571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:32.657082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:39157)
0926 22:43:32.657201 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:32.657249 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260926 22:43:32.788957 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:32.788757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:58631)
0926 22:43:32.788871 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:32.788907 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260926 22:43:32.954739 28573 raft_consensus.cc:482] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:32.954890 28573 raft_consensus.cc:504] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:32.955246 28573 leader_election.cc:288] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173), 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841)
Sep 26 22:43:32 dist-test-slave-j8sq krb5kdc[26996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.195@KRBTEST.COM
Sep 26 22:43:32 dist-test-slave-j8sq krb5kdc[26996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.194@KRBTEST.COM
I20260926 22:43:32.969892 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:32.955682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:54567 (local address 127.11.107.194:34173)
0926 22:43:32.955830 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:32.955838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:32.955862 (+ 24us) server_negotiation.cc:407] Connection header received
0926 22:43:32.956264 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:32.956266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:32.956301 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:32.956454 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:32.957162 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:32.958066 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:32.959024 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:32.959538 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:32.959643 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:32.966752 (+ 7109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:32.966755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:32.966756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:32.966771 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:32.968339 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:32.968701 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:32.968704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:32.968704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:32.968799 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:32.969022 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:32.969023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:32.969023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:32.969257 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:32.969349 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:32.969631 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 22:43:32.969749 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260926 22:43:32.969918 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:32.955546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:53967 (local address 127.11.107.195:33841)
0926 22:43:32.955712 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:32.955726 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:32.955762 (+ 36us) server_negotiation.cc:407] Connection header received
0926 22:43:32.956229 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:32.956231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:32.956273 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:32.956380 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:32.957169 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:32.958052 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:32.959033 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:32.959526 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:32.959648 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:32.966235 (+ 6587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:32.966240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:32.966240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:32.966253 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:32.968163 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:32.968625 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:32.968629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:32.968629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:32.968742 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:32.969022 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:32.969023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:32.969023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:32.969260 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:32.969349 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:32.969632 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 22:43:32.969781 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":59,"threads_started":1}
I20260926 22:43:32.969936 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:32.955440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:33841 (local address 127.11.107.193:53967)
0926 22:43:32.955499 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:32.955512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:32.955583 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:32.956456 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:32.956459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:32.956856 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:32.957036 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:32.957041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:32.958141 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:32.958142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:32.958924 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:32.958926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:32.959629 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:32.959630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:32.959748 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:32.964285 (+ 4537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:32.964296 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:32.966129 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:32.968245 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:32.968252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:32.968254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:32.968545 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:32.968820 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:32.968821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:32.968821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:32.968953 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:32.969343 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:32.969344 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:32.969468 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 22:43:32.969565 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 22:43:32.969747 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":32}
I20260926 22:43:32.969939 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:32.955459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:34173 (local address 127.11.107.193:54567)
0926 22:43:32.955555 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:32.955566 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:32.955619 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:32.956463 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:32.956464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:32.956855 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:32.957036 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:32.957041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:32.958175 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:32.958176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:32.958924 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:32.958927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:32.959731 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:32.959732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:32.959778 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:32.964285 (+ 4507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:32.964296 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:32.966639 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:32.968418 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:32.968419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:32.968419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:32.968625 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:32.968871 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:32.968872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:32.968872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:32.968952 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:32.969331 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:32.969338 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:32.969466 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 22:43:32.969574 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 22:43:32.969801 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":52,"thread_start_us":48,"threads_started":1}
W20260926 22:43:32.970790 28188 leader_election.cc:341] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841): Not found: Tablet not found: b0f60f20f15342638c0edcfe2c759b05
W20260926 22:43:32.971211 28187 leader_election.cc:341] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173): Not found: Tablet not found: b0f60f20f15342638c0edcfe2c759b05
I20260926 22:43:32.971266 28187 leader_election.cc:302] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12a5559fe9574cdeaecea34832279b54; no voters: 477ee41f35fa4b91a2361be5bbeb098b, 91f3775fd6e747dc806b4fe0b8e8ec72
I20260926 22:43:32.971374 28573 raft_consensus.cc:2726] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:43:33.603046 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:33.602849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:41881)
0926 22:43:33.602965 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:33.603000 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260926 22:43:33.658005 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:33.657748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:48133)
0926 22:43:33.657937 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:33.657970 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
I20260926 22:43:33.789580 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:33.789374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:46055)
0926 22:43:33.789503 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:33.789547 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260926 22:43:34.568991 28580 raft_consensus.cc:482] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:34.569077 28580 raft_consensus.cc:504] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:34.569228 28580 leader_election.cc:288] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173), 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841)
W20260926 22:43:34.569664 28187 leader_election.cc:341] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173): Not found: Tablet not found: b0f60f20f15342638c0edcfe2c759b05
W20260926 22:43:34.569772 28188 leader_election.cc:341] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841): Not found: Tablet not found: b0f60f20f15342638c0edcfe2c759b05
I20260926 22:43:34.569806 28188 leader_election.cc:302] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12a5559fe9574cdeaecea34832279b54; no voters: 477ee41f35fa4b91a2361be5bbeb098b, 91f3775fd6e747dc806b4fe0b8e8ec72
I20260926 22:43:34.569902 28580 raft_consensus.cc:2726] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:43:34.603727 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:34.603497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:38845)
0926 22:43:34.603631 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:34.603667 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260926 22:43:34.658783 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:34.658515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:56983)
0926 22:43:34.658714 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:34.658746 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":127,"threads_started":1}
I20260926 22:43:34.790280 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:34.790009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:34311)
0926 22:43:34.790144 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:34.790234 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":56,"threads_started":1}
I20260926 22:43:35.604341 28584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:35.604145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:52303)
0926 22:43:35.604261 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:35.604296 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260926 22:43:35.659449 28585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:35.659263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:51801)
0926 22:43:35.659379 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:35.659413 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
I20260926 22:43:35.790956 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:35.790761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:37785)
0926 22:43:35.790881 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:35.790919 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260926 22:43:36.459743 28587 raft_consensus.cc:482] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:36.459836 28587 raft_consensus.cc:504] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:36.460019 28587 leader_election.cc:288] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173), 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841)
W20260926 22:43:36.460384 28187 leader_election.cc:341] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173): Not found: Tablet not found: b0f60f20f15342638c0edcfe2c759b05
W20260926 22:43:36.460536 28188 leader_election.cc:341] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841): Not found: Tablet not found: b0f60f20f15342638c0edcfe2c759b05
I20260926 22:43:36.460623 28188 leader_election.cc:302] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12a5559fe9574cdeaecea34832279b54; no voters: 477ee41f35fa4b91a2361be5bbeb098b, 91f3775fd6e747dc806b4fe0b8e8ec72
I20260926 22:43:36.460709 28587 raft_consensus.cc:2726] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:43:36.605067 28588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:36.604837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:45523)
0926 22:43:36.604987 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:36.605021 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
I20260926 22:43:36.660068 28589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:36.659898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:36537)
0926 22:43:36.660001 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:36.660029 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260926 22:43:36.791604 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:36.791414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:51413)
0926 22:43:36.791523 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:36.791561 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260926 22:43:37.605777 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:37.605535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:34735)
0926 22:43:37.605697 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:37.605731 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260926 22:43:37.660770 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:37.660554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:49843)
0926 22:43:37.660697 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:37.660731 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260926 22:43:37.792253 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:37.792045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:50137)
0926 22:43:37.792176 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:37.792216 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260926 22:43:38.606451 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:38.606207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:44177)
0926 22:43:38.606359 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:38.606392 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260926 22:43:38.661454 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:38.661221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:54449)
0926 22:43:38.661367 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:38.661413 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
I20260926 22:43:38.792999 28596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:38.792795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:60987)
0926 22:43:38.792923 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:38.792960 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260926 22:43:38.868700 28597 raft_consensus.cc:482] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:38.868795 28597 raft_consensus.cc:504] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:38.869015 28597 leader_election.cc:288] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173), 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841)
W20260926 22:43:38.869488 28188 leader_election.cc:341] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841): Not found: Tablet not found: b0f60f20f15342638c0edcfe2c759b05
W20260926 22:43:38.869572 28187 leader_election.cc:341] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173): Not found: Tablet not found: b0f60f20f15342638c0edcfe2c759b05
I20260926 22:43:38.869661 28187 leader_election.cc:302] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12a5559fe9574cdeaecea34832279b54; no voters: 477ee41f35fa4b91a2361be5bbeb098b, 91f3775fd6e747dc806b4fe0b8e8ec72
I20260926 22:43:38.869761 28597 raft_consensus.cc:2726] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:43:39.607136 28598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:39.606918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:34775)
0926 22:43:39.607043 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:39.607084 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260926 22:43:39.662084 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:39.661898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:47653)
0926 22:43:39.662015 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:39.662047 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260926 22:43:39.793771 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:39.793469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:57955)
0926 22:43:39.793694 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:39.793731 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":74,"threads_started":1}
I20260926 22:43:40.607919 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:40.607659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:36635)
0926 22:43:40.607827 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:40.607869 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260926 22:43:40.662745 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:40.662501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:43973)
0926 22:43:40.662646 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:40.662705 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260926 22:43:40.794497 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:40.794290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:42963)
0926 22:43:40.794403 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:40.794437 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260926 22:43:41.608668 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:41.608432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:57027)
0926 22:43:41.608568 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:41.608602 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260926 22:43:41.663554 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:41.663347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:55791)
0926 22:43:41.663472 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:41.663509 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260926 22:43:41.795130 28606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:41.794936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:56113)
0926 22:43:41.795049 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:41.795081 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260926 22:43:42.609367 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:42.609166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:44649)
0926 22:43:42.609288 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:42.609321 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260926 22:43:42.664295 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:42.664062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:52297)
0926 22:43:42.664206 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:42.664245 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260926 22:43:42.795786 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:42.795587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:44839)
0926 22:43:42.795710 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:42.795743 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260926 22:43:43.237257 28610 raft_consensus.cc:482] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:43.237341 28610 raft_consensus.cc:504] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:43.237545 28610 leader_election.cc:288] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173), 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841)
W20260926 22:43:43.237920 28187 leader_election.cc:341] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173): Not found: Tablet not found: b0f60f20f15342638c0edcfe2c759b05
W20260926 22:43:43.238009 28188 leader_election.cc:341] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841): Not found: Tablet not found: b0f60f20f15342638c0edcfe2c759b05
I20260926 22:43:43.238070 28188 leader_election.cc:302] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12a5559fe9574cdeaecea34832279b54; no voters: 477ee41f35fa4b91a2361be5bbeb098b, 91f3775fd6e747dc806b4fe0b8e8ec72
I20260926 22:43:43.238157 28610 raft_consensus.cc:2726] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:43:43.610148 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:43.609932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:33163)
0926 22:43:43.610065 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:43.610101 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":54,"threads_started":1}
I20260926 22:43:43.664906 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:43.664705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:47515)
0926 22:43:43.664822 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:43.664854 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260926 22:43:43.796419 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:43.796211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:46417)
0926 22:43:43.796340 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:43.796375 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260926 22:43:44.610817 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:44.610616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:56239)
0926 22:43:44.610737 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:44.610772 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260926 22:43:44.665464 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:44.665280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:60089)
0926 22:43:44.665390 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:44.665421 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260926 22:43:44.797128 28616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:44.796912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:39505)
0926 22:43:44.797038 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:44.797076 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260926 22:43:45.611500 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:45.611289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:38277)
0926 22:43:45.611411 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:45.611452 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260926 22:43:45.666141 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:45.665937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:46513)
0926 22:43:45.666057 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:45.666097 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260926 22:43:45.797861 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:45.797606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:58909)
0926 22:43:45.797759 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:45.797795 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260926 22:43:46.304757 28620 raft_consensus.cc:482] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:46.304870 28620 raft_consensus.cc:504] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "477ee41f35fa4b91a2361be5bbeb098b" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34173 } } peers { permanent_uuid: "91f3775fd6e747dc806b4fe0b8e8ec72" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 33841 } } peers { permanent_uuid: "12a5559fe9574cdeaecea34832279b54" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 43167 } }
I20260926 22:43:46.305075 28620 leader_election.cc:288] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173), 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841)
W20260926 22:43:46.305498 28188 leader_election.cc:341] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91f3775fd6e747dc806b4fe0b8e8ec72 (127.11.107.195:33841): Not found: Tablet not found: b0f60f20f15342638c0edcfe2c759b05
W20260926 22:43:46.305593 28187 leader_election.cc:341] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 477ee41f35fa4b91a2361be5bbeb098b (127.11.107.194:34173): Not found: Tablet not found: b0f60f20f15342638c0edcfe2c759b05
I20260926 22:43:46.305665 28187 leader_election.cc:302] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12a5559fe9574cdeaecea34832279b54; no voters: 477ee41f35fa4b91a2361be5bbeb098b, 91f3775fd6e747dc806b4fe0b8e8ec72
I20260926 22:43:46.305754 28620 raft_consensus.cc:2726] T b0f60f20f15342638c0edcfe2c759b05 P 12a5559fe9574cdeaecea34832279b54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:43:46.612258 28621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:46.612030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.193:44985)
0926 22:43:46.612170 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:46.612210 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260926 22:43:46.666854 28622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:46.666652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.194:58943)
0926 22:43:46.666781 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:46.666812 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260926 22:43:46.798535 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:46.798346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45429 (local address 127.11.107.195:49219)
0926 22:43:46.798463 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:46.798495 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.254:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260926 22:43:46.803980 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 28137
I20260926 22:43:46.808908 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 28304
I20260926 22:43:46.813017 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 28437
2026-09-26T22:43:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58208 ms)
[----------] 30 tests from SecurityITest (225869 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:43:46 dist-test-slave-j8sq krb5kdc[28626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:43:46 dist-test-slave-j8sq krb5kdc[28626](info): set up 2 sockets
Sep 26 22:43:46 dist-test-slave-j8sq krb5kdc[28626](info): commencing operation
krb5kdc: starting...
W20260926 22:43:47.870390 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:43:47 dist-test-slave-j8sq krb5kdc[28626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:43:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:43:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:43:47.987320 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:44897
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:44897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:48.067909 28639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:48.068121 28639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:48.068145 28639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:48.069321 28639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:43:48.069365 28639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:48.069379 28639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:48.069391 28639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:43:48.069403 28639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:43:48.070971 28639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:44897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:44897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:48.071209 28639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:48.071449 28639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:48.076128 28644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:48.076156 28647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:48.076156 28645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:48.076426 28639 server_base.cc:1034] running on GCE node
I20260926 22:43:48.076660 28639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:48.076952 28639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:48.078111 28639 hybrid_clock.cc:648] HybridClock initialized: now 1790462628078095 us; error 46 us; skew 500 ppm
Sep 26 22:43:48 dist-test-slave-j8sq krb5kdc[28626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462628, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:48.080565 28639 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:43:48.081072 28639 webserver.cc:460] Webserver started at http://127.11.107.254:41313/ using document root <none> and password file <none>
I20260926 22:43:48.081254 28639 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:48.081293 28639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:48.081380 28639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:48.082427 28639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "7902ea0a297e4e82bd69e66ebe08608f"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "473d02d549fd72b5bb1abe3fe4d36ed5"
server_key_iv: "fa93a7ee78394ab4de96b9d1aa8acd77"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.082739 28639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "7902ea0a297e4e82bd69e66ebe08608f"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "473d02d549fd72b5bb1abe3fe4d36ed5"
server_key_iv: "fa93a7ee78394ab4de96b9d1aa8acd77"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.084016 28639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:43:48.084688 28654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:48.084867 28639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:43:48.084954 28639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "7902ea0a297e4e82bd69e66ebe08608f"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "473d02d549fd72b5bb1abe3fe4d36ed5"
server_key_iv: "fa93a7ee78394ab4de96b9d1aa8acd77"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.085026 28639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:48.098785 28639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:48.100986 28639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:48.101157 28639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:48.105374 28639 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:44897
I20260926 22:43:48.105422 28706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:44897 every 8 connection(s)
I20260926 22:43:48.105801 28639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:43:48.106714 28707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:48.108915 28707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f: Bootstrap starting.
I20260926 22:43:48.109485 28707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:48.109774 28707 log.cc:824] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:48.110499 28707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f: No bootstrap required, opened a new log
I20260926 22:43:48.111516 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 28639
I20260926 22:43:48.111613 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:43:48.111815 11695 external_mini_cluster.cc:1428] Setting key 6d1728ff63d7589f91309415cef944ff
I20260926 22:43:48.112545 28707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7902ea0a297e4e82bd69e66ebe08608f" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44897 } }
I20260926 22:43:48.112671 28707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:48.112694 28707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7902ea0a297e4e82bd69e66ebe08608f, State: Initialized, Role: FOLLOWER
I20260926 22:43:48.112835 28707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7902ea0a297e4e82bd69e66ebe08608f" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44897 } }
I20260926 22:43:48.112914 28707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:43:48.112970 28707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:43:48.113052 28707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:48.113579 28707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7902ea0a297e4e82bd69e66ebe08608f" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44897 } }
I20260926 22:43:48.113692 28707 leader_election.cc:302] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7902ea0a297e4e82bd69e66ebe08608f; no voters:
I20260926 22:43:48.113889 28707 leader_election.cc:288] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:43:48.113924 28712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:48.114068 28712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [term 1 LEADER]: Becoming Leader. State: Replica: 7902ea0a297e4e82bd69e66ebe08608f, State: Running, Role: LEADER
I20260926 22:43:48.114151 28707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:43:48.114171 28712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7902ea0a297e4e82bd69e66ebe08608f" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44897 } }
I20260926 22:43:48.114507 28712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7902ea0a297e4e82bd69e66ebe08608f. Latest consensus state: current_term: 1 leader_uuid: "7902ea0a297e4e82bd69e66ebe08608f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7902ea0a297e4e82bd69e66ebe08608f" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44897 } } }
I20260926 22:43:48.114519 28714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7902ea0a297e4e82bd69e66ebe08608f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7902ea0a297e4e82bd69e66ebe08608f" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 44897 } } }
I20260926 22:43:48.114687 28712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:48.114701 28714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f [sys.catalog]: This master's current role is: LEADER
W20260926 22:43:48.115906 28728 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:43:48.115963 28728 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:43:48.116006 28721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:43:48.116170 28721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:43:48.117936 28721 catalog_manager.cc:1347] Generated new cluster ID: 76e6b4c9fd37464e9397a759e2ee4458
I20260926 22:43:48.117993 28721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:43:48 dist-test-slave-j8sq krb5kdc[28626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:48.123924 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.112327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.11.107.254:44897)
0926 22:43:48.112538 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.112545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.112568 (+ 23us) server_negotiation.cc:407] Connection header received
0926 22:43:48.112704 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.112712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.112769 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.112927 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:48.113678 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.115013 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.116146 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.116755 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.116902 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.119737 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:48.119740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:48.119741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:48.119752 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:48.122343 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:48.122748 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:48.122751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:48.122751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:48.122859 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:48.123068 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:48.123069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:48.123069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:48.123345 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:48.123484 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.123611 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.123732 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
I20260926 22:43:48.132827 28721 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:43:48.133308 28721 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:43:48.140652 28721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f: Generated new TSK 0
I20260926 22:43:48.140831 28721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:43:48.170857 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44897
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:48.251012 28733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:48.251247 28733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:48.251284 28733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:48.252453 28733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:48.252516 28733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:48.252578 28733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:43:48.254145 28733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:48.254395 28733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:48.254632 28733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:48.259701 28741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:48.259783 28733 server_base.cc:1034] running on GCE node
W20260926 22:43:48.259701 28738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:48.259716 28739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:48.260222 28733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:48.260434 28733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:48.261598 28733 hybrid_clock.cc:648] HybridClock initialized: now 1790462628261562 us; error 40 us; skew 500 ppm
Sep 26 22:43:48 dist-test-slave-j8sq krb5kdc[28626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462628, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:48.264086 28733 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:43:48.264495 28733 webserver.cc:460] Webserver started at http://127.11.107.193:43345/ using document root <none> and password file <none>
I20260926 22:43:48.264719 28733 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:48.264786 28733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:48.264937 28733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:48.266093 28733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "5560ed62514b4b3dad74d8deefbd73c7"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "7b02588898a60f2c829b305ad1dfb530"
server_key_iv: "42581a4662a27ded2eb7c9fd493e9aa4"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.266443 28733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "5560ed62514b4b3dad74d8deefbd73c7"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "7b02588898a60f2c829b305ad1dfb530"
server_key_iv: "42581a4662a27ded2eb7c9fd493e9aa4"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.267858 28733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:43:48.268608 28748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:48.268906 28733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:48.269008 28733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "5560ed62514b4b3dad74d8deefbd73c7"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "7b02588898a60f2c829b305ad1dfb530"
server_key_iv: "42581a4662a27ded2eb7c9fd493e9aa4"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.269079 28733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:48.287869 28733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:48.290081 28733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:48.290248 28733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:48.290493 28733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:48.290822 28733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:48.290879 28733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:48.290926 28733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:48.290961 28733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:48.296862 28733 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:45037
I20260926 22:43:48.296904 28861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:45037 every 8 connection(s)
I20260926 22:43:48.297210 28733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
Sep 26 22:43:48 dist-test-slave-j8sq krb5kdc[28626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462628, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:48.305892 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 28733
I20260926 22:43:48.306010 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:43:48.306103 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.297915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:37339 (local address 127.11.107.254:44897)
0926 22:43:48.297994 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.297997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.298187 (+ 190us) server_negotiation.cc:407] Connection header received
0926 22:43:48.298647 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.298648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.298674 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.298724 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:48.299596 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.300088 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.301095 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.301287 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.301337 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.303548 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:48.303550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:48.303550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:48.303557 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:48.304826 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:48.305216 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:48.305217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:48.305217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:48.305277 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:48.305484 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:48.305485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:48.305485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:48.305583 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:48.305655 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.305901 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.305993 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 22:43:48.306264 11695 external_mini_cluster.cc:1428] Setting key 512872a2b28c2506a8b11a70fbf59f1a
I20260926 22:43:48.306327 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.297875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44897 (local address 127.11.107.193:37339)
0926 22:43:48.298070 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:48.298108 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:48.298557 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:48.298783 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:48.298786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:48.299340 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:48.299524 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.299528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.300189 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.300191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.301012 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.301015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.301362 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.301363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.301466 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.301965 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:48.301974 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:48.303457 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:48.304919 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:48.304925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:48.304927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:48.305152 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:48.305338 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:48.305338 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:48.305339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:48.305433 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:48.305637 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:48.305641 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:48.305747 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 22:43:48.305826 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 22:43:48.306056 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":110,"threads_started":1}
I20260926 22:43:48.306962 28862 heartbeater.cc:347] Connected to a master server at 127.11.107.254:44897
I20260926 22:43:48.307065 28862 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:48.307253 28862 heartbeater.cc:511] Master 127.11.107.254:44897 requested a full tablet report, sending...
I20260926 22:43:48.307751 28671 ts_manager.cc:194] Registered new tserver with Master: 5560ed62514b4b3dad74d8deefbd73c7 (127.11.107.193:45037)
I20260926 22:43:48.308352 28671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:37339
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:43:48.330602 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44897
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:48.412161 28867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:48.412389 28867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:48.412415 28867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:48.413481 28867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:48.413518 28867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:48.413545 28867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:43:48.415198 28867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:48.415447 28867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:48.415729 28867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:48.420815 28875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:48.420905 28872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:48.420805 28873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:48.421087 28867 server_base.cc:1034] running on GCE node
I20260926 22:43:48.421327 28867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:48.421579 28867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:48.422744 28867 hybrid_clock.cc:648] HybridClock initialized: now 1790462628422727 us; error 33 us; skew 500 ppm
Sep 26 22:43:48 dist-test-slave-j8sq krb5kdc[28626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462628, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:48.425352 28867 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:43:48.425761 28867 webserver.cc:460] Webserver started at http://127.11.107.194:36143/ using document root <none> and password file <none>
I20260926 22:43:48.426005 28867 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:48.426070 28867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:48.426210 28867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:48.427232 28867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "8b566d913b6a4d1d9589231498d66f3a"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "14695172a66b85a4369feb5f09171960"
server_key_iv: "3c5ffbc5385173ad96b3b319c72c2401"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.427611 28867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "8b566d913b6a4d1d9589231498d66f3a"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "14695172a66b85a4369feb5f09171960"
server_key_iv: "3c5ffbc5385173ad96b3b319c72c2401"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.428885 28867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:43:48.429574 28882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:48.429756 28867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:43:48.429865 28867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "8b566d913b6a4d1d9589231498d66f3a"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "14695172a66b85a4369feb5f09171960"
server_key_iv: "3c5ffbc5385173ad96b3b319c72c2401"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.429949 28867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:48.441807 28867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:48.443527 28867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:48.443629 28867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:48.443785 28867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:48.444001 28867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:48.444029 28867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:48.444049 28867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:48.444061 28867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:48.449366 28867 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:34591
I20260926 22:43:48.449456 28995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:34591 every 8 connection(s)
I20260926 22:43:48.449710 28867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:43:48.455024 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 28867
I20260926 22:43:48.455117 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:43:48.455302 11695 external_mini_cluster.cc:1428] Setting key 3e437b588c41af8e1cb5c175233d334a
Sep 26 22:43:48 dist-test-slave-j8sq krb5kdc[28626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462628, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:48.459082 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.450428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:53181 (local address 127.11.107.254:44897)
0926 22:43:48.450503 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.450505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.450673 (+ 168us) server_negotiation.cc:407] Connection header received
0926 22:43:48.451105 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.451106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.451117 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.451182 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:48.452045 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.452571 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.453508 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.453727 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.453792 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.456006 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:48.456008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:48.456008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:48.456016 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:48.457683 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:48.458117 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:48.458118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:48.458119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:48.458182 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:48.458413 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:48.458414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:48.458414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:48.458525 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:48.458581 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.458911 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.458993 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 22:43:48.459263 28998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.450403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44897 (local address 127.11.107.194:53181)
0926 22:43:48.450551 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:48.450582 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:48.451014 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:48.451233 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:48.451236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:48.451775 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:48.451945 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.451950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.452655 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.452655 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.453416 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.453418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.453812 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.453813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.453924 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.454332 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:48.454340 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:48.455841 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:48.457776 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:48.457782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:48.457783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:48.458047 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:48.458253 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:48.458253 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:48.458254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:48.458346 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:48.458620 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:48.458627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:48.458734 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 22:43:48.458889 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 22:43:48.459037 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":90,"threads_started":1}
I20260926 22:43:48.459723 28996 heartbeater.cc:347] Connected to a master server at 127.11.107.254:44897
I20260926 22:43:48.459828 28996 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:48.460026 28996 heartbeater.cc:511] Master 127.11.107.254:44897 requested a full tablet report, sending...
I20260926 22:43:48.460404 28671 ts_manager.cc:194] Registered new tserver with Master: 8b566d913b6a4d1d9589231498d66f3a (127.11.107.194:34591)
I20260926 22:43:48.460768 28671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:53181
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:43:48.480684 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44897
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:48.562916 29001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:48.563119 29001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:48.563143 29001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:48.564285 29001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:48.564328 29001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:48.564383 29001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:43:48.565814 29001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:44897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:48.566126 29001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:48.566366 29001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:48.571163 29007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:48.571224 29009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:48.571266 29006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:48.571480 29001 server_base.cc:1034] running on GCE node
I20260926 22:43:48.571672 29001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:48.571923 29001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:48.573081 29001 hybrid_clock.cc:648] HybridClock initialized: now 1790462628573066 us; error 26 us; skew 500 ppm
Sep 26 22:43:48 dist-test-slave-j8sq krb5kdc[28626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462628, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:48.575574 29001 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:43:48.576032 29001 webserver.cc:460] Webserver started at http://127.11.107.195:38881/ using document root <none> and password file <none>
I20260926 22:43:48.576248 29001 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:48.576290 29001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:48.576375 29001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:48.577320 29001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "179aa8319a1f4a94ad49a274527a81ec"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "2c4371e51f9667fc853caa662c079e7c"
server_key_iv: "64744f81e2c78c2d8f812c4f26fb07e1"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.577617 29001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "179aa8319a1f4a94ad49a274527a81ec"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "2c4371e51f9667fc853caa662c079e7c"
server_key_iv: "64744f81e2c78c2d8f812c4f26fb07e1"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.578938 29001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:43:48.579672 29016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:48.579850 29001 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:43:48.579927 29001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "179aa8319a1f4a94ad49a274527a81ec"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "2c4371e51f9667fc853caa662c079e7c"
server_key_iv: "64744f81e2c78c2d8f812c4f26fb07e1"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.579977 29001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:48.590801 29001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:48.592720 29001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:48.592836 29001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:48.593004 29001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:48.593305 29001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:48.593340 29001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:48.593361 29001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:48.593405 29001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:48.598989 29001 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:42105
I20260926 22:43:48.599061 29129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:42105 every 8 connection(s)
I20260926 22:43:48.599354 29001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:43:48.605149 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 29001
Sep 26 22:43:48 dist-test-slave-j8sq krb5kdc[28626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462628, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:48.605271 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:43:48.605479 11695 external_mini_cluster.cc:1428] Setting key 06695bcf35bc4dd6af16804c062db456
I20260926 22:43:48.608278 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.600067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:40975 (local address 127.11.107.254:44897)
0926 22:43:48.600134 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.600136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.600324 (+ 188us) server_negotiation.cc:407] Connection header received
0926 22:43:48.600838 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.600839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.600852 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.600913 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:48.601805 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.602331 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.603310 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.603524 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.603615 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.605885 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:48.605886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:48.605886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:48.605893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:48.607168 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:48.607562 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:48.607563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:48.607563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:48.607611 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:48.607784 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:48.607785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:48.607785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:48.607880 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:48.607930 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.608161 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.608216 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 22:43:48.608530 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.600055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:44897 (local address 127.11.107.195:40975)
0926 22:43:48.600214 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:48.600245 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:48.600730 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:48.600982 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:48.600989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:48.601522 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:48.601703 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.601708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.602424 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.602425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.603215 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.603217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.603623 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.603625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.603741 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.604219 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:48.604226 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:48.605750 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:48.607277 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:48.607282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:48.607284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:48.607498 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:48.607651 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:48.607652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:48.607652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:48.607735 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:48.607936 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:48.607940 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:48.608046 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 22:43:48.608136 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 22:43:48.608245 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260926 22:43:48.608970 29130 heartbeater.cc:347] Connected to a master server at 127.11.107.254:44897
I20260926 22:43:48.609061 29130 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:48.609232 29130 heartbeater.cc:511] Master 127.11.107.254:44897 requested a full tablet report, sending...
I20260926 22:43:48.609601 28671 ts_manager.cc:194] Registered new tserver with Master: 179aa8319a1f4a94ad49a274527a81ec (127.11.107.195:42105)
I20260926 22:43:48.610054 28671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:40975
I20260926 22:43:48.617524 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:43:48.624473 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.619019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.11.107.254:44897)
0926 22:43:48.619068 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.619070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.619126 (+ 56us) server_negotiation.cc:407] Connection header received
0926 22:43:48.619176 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.619177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.619184 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.619231 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:48.619779 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.620243 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.620954 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.621121 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.621166 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.622284 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:48.622285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:48.622285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:48.622291 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:48.623485 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:48.623836 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:48.623838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:48.623838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:48.623900 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:48.624070 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:48.624071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:48.624071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:48.624213 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:48.624285 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.624365 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.624417 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:43:48.626282 28671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:43:48.627835 28671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:43:48.633762 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.629884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:34591 (local address 127.0.0.1:46092)
0926 22:43:48.630035 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:48.630079 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:48.630139 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:48.630342 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:48.630347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:48.630349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:48.630414 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.630416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.631653 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.631654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.632629 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.632632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.633435 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.633436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.633516 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.633570 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 22:43:48.633639 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 22:43:48.633693 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260926 22:43:48.633775 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.629964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.11.107.194:34591)
0926 22:43:48.630104 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.630109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.630142 (+ 33us) server_negotiation.cc:407] Connection header received
0926 22:43:48.630233 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.630235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.630247 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.630338 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:48.630502 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.631562 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.632755 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.633354 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.633437 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.633494 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.633653 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.633692 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260926 22:43:48.634676 29141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.630330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:42105 (local address 127.0.0.1:53424)
0926 22:43:48.630416 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:48.630426 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:48.630506 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:48.630758 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:48.630759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:48.630762 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:48.630811 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.630813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.632234 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.632235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.633470 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.633473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.634449 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.634450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.634513 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.634517 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:43:48.634575 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 22:43:48.634611 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":52,"threads_started":1}
I20260926 22:43:48.634747 29140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.630156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.11.107.195:42105)
0926 22:43:48.630319 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.630324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.630476 (+ 152us) server_negotiation.cc:407] Connection header received
0926 22:43:48.630643 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.630646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.630662 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.630741 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:48.631008 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.632109 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.633597 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.634355 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.634450 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.634500 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.634606 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.634657 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260926 22:43:48.634773 28930 tablet_service.cc:1467] Processing CreateTablet for tablet 54e9b05d67cf467da7274a7cca20e333 (DEFAULT_TABLE table=test-table [id=2b414d81c525430f90a80bea963600c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:48.635011 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.630494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:45037 (local address 127.0.0.1:57862)
0926 22:43:48.630867 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:48.630878 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:48.630923 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:48.631257 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:48.631258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:48.631259 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:48.631303 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.631304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.632661 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.632662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.633720 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.633722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.634757 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.634758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.634821 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.634826 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:43:48.634888 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 22:43:48.634938 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":43,"threads_started":1}
I20260926 22:43:48.635134 28930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e9b05d67cf467da7274a7cca20e333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:48.635294 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.630675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.11.107.193:45037)
0926 22:43:48.631085 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.631090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.631099 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:43:48.631140 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.631142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.631158 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.631247 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:48.631371 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.632408 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.633822 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.634590 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.634664 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.634716 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.635129 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.635177 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":109,"threads_started":1}
I20260926 22:43:48.635545 29064 tablet_service.cc:1467] Processing CreateTablet for tablet 54e9b05d67cf467da7274a7cca20e333 (DEFAULT_TABLE table=test-table [id=2b414d81c525430f90a80bea963600c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:48.635784 29064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e9b05d67cf467da7274a7cca20e333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:48.637648 29144 tablet_bootstrap.cc:492] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a: Bootstrap starting.
I20260926 22:43:48.638375 29144 tablet_bootstrap.cc:654] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:48.638372 29145 tablet_bootstrap.cc:492] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec: Bootstrap starting.
I20260926 22:43:48.638948 29144 log.cc:824] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:48.639025 28796 tablet_service.cc:1467] Processing CreateTablet for tablet 54e9b05d67cf467da7274a7cca20e333 (DEFAULT_TABLE table=test-table [id=2b414d81c525430f90a80bea963600c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:48.639151 29145 tablet_bootstrap.cc:654] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:48.639370 28796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e9b05d67cf467da7274a7cca20e333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:48.639709 29145 log.cc:824] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:48.639736 29144 tablet_bootstrap.cc:492] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a: No bootstrap required, opened a new log
I20260926 22:43:48.639818 29144 ts_tablet_manager.cc:1397] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:48.640447 29145 tablet_bootstrap.cc:492] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec: No bootstrap required, opened a new log
I20260926 22:43:48.640533 29145 ts_tablet_manager.cc:1397] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:48.641508 29144 raft_consensus.cc:348] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b566d913b6a4d1d9589231498d66f3a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34591 } } peers { permanent_uuid: "179aa8319a1f4a94ad49a274527a81ec" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 42105 } } peers { permanent_uuid: "5560ed62514b4b3dad74d8deefbd73c7" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45037 } }
I20260926 22:43:48.641688 29144 raft_consensus.cc:374] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:48.641736 29144 raft_consensus.cc:729] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b566d913b6a4d1d9589231498d66f3a, State: Initialized, Role: FOLLOWER
I20260926 22:43:48.641893 29144 consensus_queue.cc:260] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b566d913b6a4d1d9589231498d66f3a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34591 } } peers { permanent_uuid: "179aa8319a1f4a94ad49a274527a81ec" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 42105 } } peers { permanent_uuid: "5560ed62514b4b3dad74d8deefbd73c7" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45037 } }
I20260926 22:43:48.642242 28996 heartbeater.cc:503] Master 127.11.107.254:44897 was elected leader, sending a full tablet report...
I20260926 22:43:48.642618 29149 tablet_bootstrap.cc:492] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7: Bootstrap starting.
I20260926 22:43:48.642215 29144 ts_tablet_manager.cc:1428] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 22:43:48.642654 29145 raft_consensus.cc:348] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b566d913b6a4d1d9589231498d66f3a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34591 } } peers { permanent_uuid: "179aa8319a1f4a94ad49a274527a81ec" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 42105 } } peers { permanent_uuid: "5560ed62514b4b3dad74d8deefbd73c7" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45037 } }
I20260926 22:43:48.642804 29145 raft_consensus.cc:374] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:48.642849 29145 raft_consensus.cc:729] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179aa8319a1f4a94ad49a274527a81ec, State: Initialized, Role: FOLLOWER
I20260926 22:43:48.642956 29145 consensus_queue.cc:260] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b566d913b6a4d1d9589231498d66f3a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34591 } } peers { permanent_uuid: "179aa8319a1f4a94ad49a274527a81ec" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 42105 } } peers { permanent_uuid: "5560ed62514b4b3dad74d8deefbd73c7" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45037 } }
I20260926 22:43:48.643189 29145 ts_tablet_manager.cc:1428] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:43:48.643357 29130 heartbeater.cc:503] Master 127.11.107.254:44897 was elected leader, sending a full tablet report...
I20260926 22:43:48.643420 29149 tablet_bootstrap.cc:654] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:48.644047 29149 log.cc:824] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:48.644696 29149 tablet_bootstrap.cc:492] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7: No bootstrap required, opened a new log
I20260926 22:43:48.644783 29149 ts_tablet_manager.cc:1397] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:48.646377 29149 raft_consensus.cc:348] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b566d913b6a4d1d9589231498d66f3a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34591 } } peers { permanent_uuid: "179aa8319a1f4a94ad49a274527a81ec" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 42105 } } peers { permanent_uuid: "5560ed62514b4b3dad74d8deefbd73c7" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45037 } }
I20260926 22:43:48.646535 29149 raft_consensus.cc:374] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:48.646595 29149 raft_consensus.cc:729] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5560ed62514b4b3dad74d8deefbd73c7, State: Initialized, Role: FOLLOWER
I20260926 22:43:48.646704 29149 consensus_queue.cc:260] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b566d913b6a4d1d9589231498d66f3a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34591 } } peers { permanent_uuid: "179aa8319a1f4a94ad49a274527a81ec" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 42105 } } peers { permanent_uuid: "5560ed62514b4b3dad74d8deefbd73c7" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45037 } }
I20260926 22:43:48.647040 29149 ts_tablet_manager.cc:1428] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:48.647082 28862 heartbeater.cc:503] Master 127.11.107.254:44897 was elected leader, sending a full tablet report...
I20260926 22:43:48.690317 29152 raft_consensus.cc:482] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:48.690452 29152 raft_consensus.cc:504] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b566d913b6a4d1d9589231498d66f3a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34591 } } peers { permanent_uuid: "179aa8319a1f4a94ad49a274527a81ec" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 42105 } } peers { permanent_uuid: "5560ed62514b4b3dad74d8deefbd73c7" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45037 } }
I20260926 22:43:48.690816 29152 leader_election.cc:288] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b566d913b6a4d1d9589231498d66f3a (127.11.107.194:34591), 179aa8319a1f4a94ad49a274527a81ec (127.11.107.195:42105)
I20260926 22:43:48.692418 29150 raft_consensus.cc:482] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:48.692531 29150 raft_consensus.cc:504] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b566d913b6a4d1d9589231498d66f3a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34591 } } peers { permanent_uuid: "179aa8319a1f4a94ad49a274527a81ec" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 42105 } } peers { permanent_uuid: "5560ed62514b4b3dad74d8deefbd73c7" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45037 } }
I20260926 22:43:48.692920 29150 leader_election.cc:288] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b566d913b6a4d1d9589231498d66f3a (127.11.107.194:34591), 5560ed62514b4b3dad74d8deefbd73c7 (127.11.107.193:45037)
I20260926 22:43:48.694679 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.691067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:42885 (local address 127.11.107.194:34591)
0926 22:43:48.691115 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.691117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.691347 (+ 230us) server_negotiation.cc:407] Connection header received
0926 22:43:48.691390 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.691391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.691398 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.691573 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:48.691731 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.692418 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.693740 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.694316 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.694362 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.694369 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.694578 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.694614 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:43:48.694710 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.691020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:42105 (local address 127.11.107.193:34689)
0926 22:43:48.691095 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:48.691108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:48.691163 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:48.691308 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:48.691309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:48.691311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:48.691389 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.691391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.692449 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.692451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.693641 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.693644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.694453 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.694454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.694537 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.694541 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:43:48.694588 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:43:48.694632 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 22:43:48.694898 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.691021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:34591 (local address 127.11.107.193:42885)
0926 22:43:48.691239 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:48.691274 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:48.691330 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:48.691605 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:48.691606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:48.691607 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:48.691660 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.691661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.692500 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.692501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.693651 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.693653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.694431 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.694432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.694500 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.694503 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:43:48.694752 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 22:43:48.694793 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":12,"thread_start_us":130,"threads_started":1}
I20260926 22:43:48.694967 29140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.691125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:34689 (local address 127.11.107.195:42105)
0926 22:43:48.691168 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.691170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.691200 (+ 30us) server_negotiation.cc:407] Connection header received
0926 22:43:48.691223 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.691224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.691231 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.691271 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:48.691562 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.692360 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.693726 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.694352 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.694416 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.694424 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.694866 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.694911 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 22:43:48.695186 29084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e9b05d67cf467da7274a7cca20e333" candidate_uuid: "5560ed62514b4b3dad74d8deefbd73c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179aa8319a1f4a94ad49a274527a81ec" is_pre_election: true
I20260926 22:43:48.695205 28950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e9b05d67cf467da7274a7cca20e333" candidate_uuid: "5560ed62514b4b3dad74d8deefbd73c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b566d913b6a4d1d9589231498d66f3a" is_pre_election: true
I20260926 22:43:48.695274 29084 raft_consensus.cc:2454] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5560ed62514b4b3dad74d8deefbd73c7 in term 0.
I20260926 22:43:48.695338 28950 raft_consensus.cc:2454] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5560ed62514b4b3dad74d8deefbd73c7 in term 0.
I20260926 22:43:48.695457 28750 leader_election.cc:302] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179aa8319a1f4a94ad49a274527a81ec, 5560ed62514b4b3dad74d8deefbd73c7; no voters:
I20260926 22:43:48.695605 29152 raft_consensus.cc:2781] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:43:48.695658 29152 raft_consensus.cc:482] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:43:48.695679 29152 raft_consensus.cc:3037] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:48.696605 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.693120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:45037 (local address 127.11.107.195:56275)
0926 22:43:48.693463 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:48.693471 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:48.693524 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:48.693704 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:48.693705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:48.693707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:48.693766 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.693767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.694612 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.694613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.695735 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.695737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.696340 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.696341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.696406 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.696411 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:43:48.696469 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 22:43:48.696511 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":86,"threads_started":1}
I20260926 22:43:48.696586 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.693385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:56275 (local address 127.11.107.193:45037)
0926 22:43:48.693419 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.693421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.693506 (+ 85us) server_negotiation.cc:407] Connection header received
0926 22:43:48.693596 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.693597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.693605 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.693633 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:48.693862 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.694513 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.695832 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.696243 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.696327 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.696334 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.696488 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.696529 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:43:48.696583 29152 raft_consensus.cc:504] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b566d913b6a4d1d9589231498d66f3a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34591 } } peers { permanent_uuid: "179aa8319a1f4a94ad49a274527a81ec" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 42105 } } peers { permanent_uuid: "5560ed62514b4b3dad74d8deefbd73c7" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45037 } }
I20260926 22:43:48.696841 29084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e9b05d67cf467da7274a7cca20e333" candidate_uuid: "5560ed62514b4b3dad74d8deefbd73c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179aa8319a1f4a94ad49a274527a81ec"
I20260926 22:43:48.696872 28950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e9b05d67cf467da7274a7cca20e333" candidate_uuid: "5560ed62514b4b3dad74d8deefbd73c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b566d913b6a4d1d9589231498d66f3a"
I20260926 22:43:48.696918 29084 raft_consensus.cc:3037] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:48.696902 28816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e9b05d67cf467da7274a7cca20e333" candidate_uuid: "179aa8319a1f4a94ad49a274527a81ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5560ed62514b4b3dad74d8deefbd73c7" is_pre_election: true
I20260926 22:43:48.696951 28950 raft_consensus.cc:3037] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:48.696993 28816 raft_consensus.cc:2379] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 179aa8319a1f4a94ad49a274527a81ec in current term 1: Already voted for candidate 5560ed62514b4b3dad74d8deefbd73c7 in this term.
I20260926 22:43:48.697546 29152 leader_election.cc:288] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [CANDIDATE]: Term 1 election: Requested vote from peers 8b566d913b6a4d1d9589231498d66f3a (127.11.107.194:34591), 179aa8319a1f4a94ad49a274527a81ec (127.11.107.195:42105)
I20260926 22:43:48.697749 29084 raft_consensus.cc:2454] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5560ed62514b4b3dad74d8deefbd73c7 in term 1.
I20260926 22:43:48.697762 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.693064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:34591 (local address 127.11.107.195:41705)
0926 22:43:48.693243 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:48.693253 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:48.693288 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:48.693468 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:48.693469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:48.693471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:48.693522 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.693523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.694464 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.694465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.696126 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:48.696129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.697489 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.697490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:48.697562 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.697565 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:43:48.697635 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 22:43:48.697681 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":598}
I20260926 22:43:48.697924 28750 leader_election.cc:302] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179aa8319a1f4a94ad49a274527a81ec, 5560ed62514b4b3dad74d8deefbd73c7; no voters:
I20260926 22:43:48.698032 29152 raft_consensus.cc:2781] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:48.698117 28950 raft_consensus.cc:2454] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5560ed62514b4b3dad74d8deefbd73c7 in term 1.
I20260926 22:43:48.698145 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.693156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:41705 (local address 127.11.107.194:34591)
0926 22:43:48.693337 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.693339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.693347 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:43:48.693367 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.693367 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.693375 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.693574 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:48.693600 (+ 26us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.694378 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.696225 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.697379 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.697497 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.697504 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.697720 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.697763 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260926 22:43:48.698179 29152 raft_consensus.cc:686] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 1 LEADER]: Becoming Leader. State: Replica: 5560ed62514b4b3dad74d8deefbd73c7, State: Running, Role: LEADER
I20260926 22:43:48.698268 29152 consensus_queue.cc:237] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b566d913b6a4d1d9589231498d66f3a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34591 } } peers { permanent_uuid: "179aa8319a1f4a94ad49a274527a81ec" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 42105 } } peers { permanent_uuid: "5560ed62514b4b3dad74d8deefbd73c7" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45037 } }
I20260926 22:43:48.698309 28950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e9b05d67cf467da7274a7cca20e333" candidate_uuid: "179aa8319a1f4a94ad49a274527a81ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b566d913b6a4d1d9589231498d66f3a" is_pre_election: true
I20260926 22:43:48.698402 28950 raft_consensus.cc:2379] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 179aa8319a1f4a94ad49a274527a81ec in current term 1: Already voted for candidate 5560ed62514b4b3dad74d8deefbd73c7 in this term.
I20260926 22:43:48.698649 29019 leader_election.cc:302] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 179aa8319a1f4a94ad49a274527a81ec; no voters: 5560ed62514b4b3dad74d8deefbd73c7, 8b566d913b6a4d1d9589231498d66f3a
I20260926 22:43:48.698786 29150 raft_consensus.cc:2726] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 22:43:48.699149 28671 catalog_manager.cc:5495] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5560ed62514b4b3dad74d8deefbd73c7 (127.11.107.193). New cstate: current_term: 1 leader_uuid: "5560ed62514b4b3dad74d8deefbd73c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b566d913b6a4d1d9589231498d66f3a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "179aa8319a1f4a94ad49a274527a81ec" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 42105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5560ed62514b4b3dad74d8deefbd73c7" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 45037 } health_report { overall_health: HEALTHY } } }
W20260926 22:43:48.700234 28997 tablet.cc:2369] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:43:48.706319 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.703919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.11.107.193:45037)
0926 22:43:48.703980 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.703982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.703990 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:43:48.704043 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.704044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.704066 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.704152 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:43:48.704291 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.704770 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.705519 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.705707 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.705790 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.705968 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:43:48.706079 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:43:48.706170 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.706188 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.706205 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 22:43:48.707926 28950 raft_consensus.cc:1261] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a [term 1 FOLLOWER]: Refusing update from remote peer 5560ed62514b4b3dad74d8deefbd73c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:48.708003 29084 raft_consensus.cc:1261] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [term 1 FOLLOWER]: Refusing update from remote peer 5560ed62514b4b3dad74d8deefbd73c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:48.708216 29152 consensus_queue.cc:1035] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b566d913b6a4d1d9589231498d66f3a" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:48.708366 29156 consensus_queue.cc:1035] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "179aa8319a1f4a94ad49a274527a81ec" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 42105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:48.711041 29160 mvcc.cc:205] Tried to move back new op lower bound from 7333734927185104896 to 7333734927148691456. Current Snapshot: MvccSnapshot[applied={T|T < 7333734927185104896}]
I20260926 22:43:48.712798 28671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:48.712877 28671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:48.713559 28671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7902ea0a297e4e82bd69e66ebe08608f: Sending DeleteTablet for 3 replicas of tablet 54e9b05d67cf467da7274a7cca20e333
I20260926 22:43:48.713938 28930 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e9b05d67cf467da7274a7cca20e333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:48 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:46092
I20260926 22:43:48.713941 29064 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e9b05d67cf467da7274a7cca20e333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:48 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:53424
I20260926 22:43:48.714161 29169 tablet_replica.cc:330] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec: stopping tablet replica
I20260926 22:43:48.714176 28796 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e9b05d67cf467da7274a7cca20e333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:48 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:57862
I20260926 22:43:48.714264 29169 raft_consensus.cc:2229] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:43:48.714375 29169 raft_consensus.cc:2258] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:48.714394 29171 tablet_replica.cc:330] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7: stopping tablet replica
I20260926 22:43:48.714416 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 28733
I20260926 22:43:48.714469 29171 raft_consensus.cc:2229] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:43:48.714551 29171 raft_consensus.cc:2258] T 54e9b05d67cf467da7274a7cca20e333 P 5560ed62514b4b3dad74d8deefbd73c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:48.714706 29170 tablet_replica.cc:330] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a: stopping tablet replica
I20260926 22:43:48.714779 29170 raft_consensus.cc:2229] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:43:48.714973 29170 raft_consensus.cc:2258] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:48.715384 29170 ts_tablet_manager.cc:1905] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:48.715725 29169 ts_tablet_manager.cc:1905] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:48.717239 29170 ts_tablet_manager.cc:1918] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:48.717299 29170 log.cc:1196] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/54e9b05d67cf467da7274a7cca20e333
I20260926 22:43:48.717832 29170 ts_tablet_manager.cc:1939] T 54e9b05d67cf467da7274a7cca20e333 P 8b566d913b6a4d1d9589231498d66f3a: Deleting consensus metadata
I20260926 22:43:48.717924 29169 ts_tablet_manager.cc:1918] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:48.717983 29169 log.cc:1196] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/54e9b05d67cf467da7274a7cca20e333
I20260926 22:43:48.718438 29169 ts_tablet_manager.cc:1939] T 54e9b05d67cf467da7274a7cca20e333 P 179aa8319a1f4a94ad49a274527a81ec: Deleting consensus metadata
I20260926 22:43:48.718492 28657 catalog_manager.cc:4841] TS 8b566d913b6a4d1d9589231498d66f3a (127.11.107.194:34591): tablet 54e9b05d67cf467da7274a7cca20e333 (table test-table [id=2b414d81c525430f90a80bea963600c5]) successfully deleted
I20260926 22:43:48.718921 28656 catalog_manager.cc:4841] TS 179aa8319a1f4a94ad49a274527a81ec (127.11.107.195:42105): tablet 54e9b05d67cf467da7274a7cca20e333 (table test-table [id=2b414d81c525430f90a80bea963600c5]) successfully deleted
W20260926 22:43:48.720420 28656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:45037 (error 108)
I20260926 22:43:48.720688 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 28867
I20260926 22:43:48.720964 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.720841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:45037 (local address 127.0.0.1:57882)
0926 22:43:48.720891 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:48.720924 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:45037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 22:43:48.722114 28656 catalog_manager.cc:4584] TS 5560ed62514b4b3dad74d8deefbd73c7 (127.11.107.193:45037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54e9b05d67cf467da7274a7cca20e333: Network error: Client connection negotiation failed: client connection to 127.11.107.193:45037: connect: Connection refused (error 111)
I20260926 22:43:48.725735 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 29001
I20260926 22:43:48.730705 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 28639
2026-09-26T22:43:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1918 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T22:43:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:43:48Z Disabled control of system clock
I20260926 22:43:48.758566 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:39603
--webserver_interface=127.11.107.254
--webserver_port=0
--builtin_ntp_servers=127.11.107.212:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:39603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:43:48.838732 29176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:48.838941 29176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:48.838963 29176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:48.840018 29176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:43:48.840062 29176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:48.840077 29176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:48.840091 29176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:43:48.840106 29176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:43:48.841384 29176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:39603
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:39603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:48.841565 29176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:48.841825 29176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:48.846570 29181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:48.846613 29182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:48.846671 29184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:48.846802 29176 server_base.cc:1034] running on GCE node
I20260926 22:43:48.847021 29176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:48.847271 29176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:48.848409 29176 hybrid_clock.cc:648] HybridClock initialized: now 1790462628848389 us; error 36 us; skew 500 ppm
I20260926 22:43:48.849608 29176 webserver.cc:460] Webserver started at http://127.11.107.254:35621/ using document root <none> and password file <none>
I20260926 22:43:48.849906 29176 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:48.849979 29176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:48.850152 29176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:48.851166 29176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "3bed74152d694a35a771f8b45ff5f0d5"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "b8f4b1ce41c4fa32273aadfc014842d7"
server_key_iv: "233158769e1ed629dafc89796240c96d"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.851516 29176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "3bed74152d694a35a771f8b45ff5f0d5"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "b8f4b1ce41c4fa32273aadfc014842d7"
server_key_iv: "233158769e1ed629dafc89796240c96d"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.852780 29176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:43:48.853482 29190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:48.853631 29176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:43:48.853742 29176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "3bed74152d694a35a771f8b45ff5f0d5"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "b8f4b1ce41c4fa32273aadfc014842d7"
server_key_iv: "233158769e1ed629dafc89796240c96d"
server_key_version: "encryptionkey@0"
I20260926 22:43:48.853826 29176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:48.868618 29176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:48.868889 29176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:48.869040 29176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:48.872949 29176 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:39603
I20260926 22:43:48.873065 29242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:39603 every 8 connection(s)
I20260926 22:43:48.873337 29176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:43:48.874408 29243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:48.876506 29243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5: Bootstrap starting.
I20260926 22:43:48.877059 29243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:48.877359 29243 log.cc:824] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:48.878095 29243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5: No bootstrap required, opened a new log
I20260926 22:43:48.879634 29243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bed74152d694a35a771f8b45ff5f0d5" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39603 } }
I20260926 22:43:48.879765 29243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:48.879787 29243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bed74152d694a35a771f8b45ff5f0d5, State: Initialized, Role: FOLLOWER
I20260926 22:43:48.879944 29243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bed74152d694a35a771f8b45ff5f0d5" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39603 } }
I20260926 22:43:48.880002 29243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:43:48.880024 29243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:43:48.880090 29243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:48.880594 29243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bed74152d694a35a771f8b45ff5f0d5" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39603 } }
I20260926 22:43:48.880684 29243 leader_election.cc:302] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bed74152d694a35a771f8b45ff5f0d5; no voters:
I20260926 22:43:48.880856 29243 leader_election.cc:288] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:43:48.880944 29246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:48.881091 29243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:43:48.881126 29246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3bed74152d694a35a771f8b45ff5f0d5, State: Running, Role: LEADER
I20260926 22:43:48.881230 29246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bed74152d694a35a771f8b45ff5f0d5" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39603 } }
I20260926 22:43:48.881564 29246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bed74152d694a35a771f8b45ff5f0d5. Latest consensus state: current_term: 1 leader_uuid: "3bed74152d694a35a771f8b45ff5f0d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bed74152d694a35a771f8b45ff5f0d5" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39603 } } }
I20260926 22:43:48.881668 29247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bed74152d694a35a771f8b45ff5f0d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bed74152d694a35a771f8b45ff5f0d5" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39603 } } }
I20260926 22:43:48.881693 29246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:48.881978 29247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:48.881996 29255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:43:48.882464 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 29176
I20260926 22:43:48.882507 29255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:43:48.882570 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:43:48.882795 11695 external_mini_cluster.cc:1428] Setting key 92de9be46beed0180d1087d62b6268fd
I20260926 22:43:48.884531 29255 catalog_manager.cc:1347] Generated new cluster ID: b50fdce2e5a545bd8bfd64a1976d0d57
I20260926 22:43:48.884585 29255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:43:48.888857 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:48.883217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.11.107.254:39603)
0926 22:43:48.883642 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:48.883650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:48.883681 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:43:48.883789 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:48.883795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:48.883860 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:48.883995 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:48.884226 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.885547 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.886620 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:48.887227 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:48.887349 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:48.888054 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:48.888059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:48.888059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:48.888272 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:48.888311 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:48.888328 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:48.888398 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:48.888472 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 22:43:48.888648 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":78,"threads_started":1}
I20260926 22:43:48.898468 29255 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:43:48.898964 29255 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:43:48.905405 29255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5: Generated new TSK 0
I20260926 22:43:48.905586 29255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:43:48.910778 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:39603
--builtin_ntp_servers=127.11.107.212:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:43:48.992004 29267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:48.992193 29267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:48.992215 29267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:48.993350 29267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:48.993389 29267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:48.993414 29267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:43:48.994753 29267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:39603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:48.994982 29267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:48.995281 29267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:49.000207 29275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:49.000288 29272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:49.000351 29267 server_base.cc:1034] running on GCE node
W20260926 22:43:49.000413 29273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:49.000612 29267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:49.000890 29267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:49.002053 29267 hybrid_clock.cc:648] HybridClock initialized: now 1790462629002036 us; error 31 us; skew 500 ppm
I20260926 22:43:49.003334 29267 webserver.cc:460] Webserver started at http://127.11.107.193:41439/ using document root <none> and password file <none>
I20260926 22:43:49.003580 29267 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:49.003648 29267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:49.003818 29267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:49.004861 29267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "33051d5add924b6dbdffc84f473be196"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "5baad031a2003fd1bbb14b8d79ea732b"
server_key_iv: "076081e28b3cc79927dffe46511b6582"
server_key_version: "encryptionkey@0"
I20260926 22:43:49.005221 29267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "33051d5add924b6dbdffc84f473be196"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "5baad031a2003fd1bbb14b8d79ea732b"
server_key_iv: "076081e28b3cc79927dffe46511b6582"
server_key_version: "encryptionkey@0"
I20260926 22:43:49.006623 29267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:43:49.007390 29281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:49.007561 29267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:43:49.007665 29267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "33051d5add924b6dbdffc84f473be196"
format_stamp: "Formatted at 2026-09-26 22:43:48 on dist-test-slave-j8sq"
server_key: "5baad031a2003fd1bbb14b8d79ea732b"
server_key_iv: "076081e28b3cc79927dffe46511b6582"
server_key_version: "encryptionkey@0"
I20260926 22:43:49.007762 29267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:49.029486 29267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:49.029824 29267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:49.030033 29267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:49.030273 29267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:49.030607 29267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:49.030661 29267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:49.030730 29267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:49.030761 29267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:49.036787 29267 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:36895
I20260926 22:43:49.036847 29394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:36895 every 8 connection(s)
I20260926 22:43:49.037192 29267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:43:49.042634 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.037993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:52805 (local address 127.11.107.254:39603)
0926 22:43:49.038063 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:49.038065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:49.038288 (+ 223us) server_negotiation.cc:407] Connection header received
0926 22:43:49.038852 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:49.038853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:49.038867 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:49.038921 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:49.039644 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.040164 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.041163 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.041378 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.041452 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.042342 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:49.042344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:49.042345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:49.042350 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:49.042365 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:49.042369 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:49.042424 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:49.042524 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 22:43:49.042554 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:43:49.042840 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.038003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:39603 (local address 127.11.107.193:52805)
0926 22:43:49.038164 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:49.038199 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:49.038740 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:49.039007 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:49.039010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:49.039019 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:49.039527 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.039532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.040263 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.040264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.041049 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.041051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.041460 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.041460 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.041588 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.042187 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:43:49.042187 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:49.042216 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:43:49.042238 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:49.042427 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:49.042432 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:49.042435 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:43:49.042516 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 22:43:49.042624 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260926 22:43:49.043344 29395 heartbeater.cc:347] Connected to a master server at 127.11.107.254:39603
I20260926 22:43:49.043437 29395 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:49.043584 29395 heartbeater.cc:511] Master 127.11.107.254:39603 requested a full tablet report, sending...
I20260926 22:43:49.044078 29207 ts_manager.cc:194] Registered new tserver with Master: 33051d5add924b6dbdffc84f473be196 (127.11.107.193:36895)
I20260926 22:43:49.044697 29207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.107.193:52805
I20260926 22:43:49.045969 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 29267
I20260926 22:43:49.046087 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:43:49.046277 11695 external_mini_cluster.cc:1428] Setting key 7180fa1b882a15fb919b61a753c05901
I20260926 22:43:49.047349 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:39603
--builtin_ntp_servers=127.11.107.212:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:43:49.129369 29398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:49.129563 29398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:49.129585 29398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:49.130776 29398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:49.130815 29398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:49.130867 29398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:43:49.132162 29398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:39603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:49.132406 29398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:49.132697 29398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:49.137377 29403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:49.137396 29404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:49.137396 29406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:49.137769 29398 server_base.cc:1034] running on GCE node
I20260926 22:43:49.138034 29398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:49.138233 29398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:49.139384 29398 hybrid_clock.cc:648] HybridClock initialized: now 1790462629139371 us; error 26 us; skew 500 ppm
I20260926 22:43:49.140661 29398 webserver.cc:460] Webserver started at http://127.11.107.194:36275/ using document root <none> and password file <none>
I20260926 22:43:49.140913 29398 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:49.140962 29398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:49.141148 29398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:49.142292 29398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "08b3c05c22394f8a8e555dcc640a3e91"
format_stamp: "Formatted at 2026-09-26 22:43:49 on dist-test-slave-j8sq"
server_key: "ccb999dcd05962230a810183a6011a75"
server_key_iv: "7172027b82ec09a779160f4a917dbd44"
server_key_version: "encryptionkey@0"
I20260926 22:43:49.142656 29398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "08b3c05c22394f8a8e555dcc640a3e91"
format_stamp: "Formatted at 2026-09-26 22:43:49 on dist-test-slave-j8sq"
server_key: "ccb999dcd05962230a810183a6011a75"
server_key_iv: "7172027b82ec09a779160f4a917dbd44"
server_key_version: "encryptionkey@0"
I20260926 22:43:49.144011 29398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:49.144814 29412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:49.145004 29398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:49.145110 29398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "08b3c05c22394f8a8e555dcc640a3e91"
format_stamp: "Formatted at 2026-09-26 22:43:49 on dist-test-slave-j8sq"
server_key: "ccb999dcd05962230a810183a6011a75"
server_key_iv: "7172027b82ec09a779160f4a917dbd44"
server_key_version: "encryptionkey@0"
I20260926 22:43:49.145198 29398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:49.155462 29398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:49.155746 29398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:49.155889 29398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:49.156090 29398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:49.156445 29398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:49.156497 29398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:49.156553 29398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:49.156577 29398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:49.162257 29398 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:35503
I20260926 22:43:49.162330 29525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:35503 every 8 connection(s)
I20260926 22:43:49.162698 29398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:43:49.167703 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.163432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:36801 (local address 127.11.107.254:39603)
0926 22:43:49.163495 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:49.163497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:49.163748 (+ 251us) server_negotiation.cc:407] Connection header received
0926 22:43:49.164216 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:49.164217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:49.164227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:49.164287 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:49.164904 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.165403 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.166400 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.166583 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.166638 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.167473 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:49.167474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:49.167474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:49.167478 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:49.167489 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:49.167493 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:49.167534 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:49.167619 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 22:43:49.167644 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:43:49.167884 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.163433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:39603 (local address 127.11.107.194:36801)
0926 22:43:49.163614 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:49.163653 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:49.164123 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:49.164338 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:49.164341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:49.164349 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:49.164832 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.164836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.165481 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.165482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.166324 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.166327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.166658 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.166659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.166765 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.167345 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:43:49.167346 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:49.167374 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:43:49.167396 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:49.167549 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:49.167554 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:49.167556 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:43:49.167607 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:43:49.167686 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260926 22:43:49.168404 29526 heartbeater.cc:347] Connected to a master server at 127.11.107.254:39603
I20260926 22:43:49.168505 29526 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:49.168689 29526 heartbeater.cc:511] Master 127.11.107.254:39603 requested a full tablet report, sending...
I20260926 22:43:49.169072 29207 ts_manager.cc:194] Registered new tserver with Master: 08b3c05c22394f8a8e555dcc640a3e91 (127.11.107.194:35503)
I20260926 22:43:49.169451 29207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.107.194:36801
I20260926 22:43:49.172407 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 29398
I20260926 22:43:49.172492 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:43:49.172703 11695 external_mini_cluster.cc:1428] Setting key e693b3f6fa73480920ab2ba98c2b305f
I20260926 22:43:49.173568 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:39603
--builtin_ntp_servers=127.11.107.212:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:43:49.253652 29529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:49.253909 29529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:49.253970 29529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:49.255122 29529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:49.255160 29529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:49.255185 29529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:43:49.256608 29529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:39603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:49.256857 29529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:49.257144 29529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:49.262214 29535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:49.262238 29537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:49.262214 29534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:49.262429 29529 server_base.cc:1034] running on GCE node
I20260926 22:43:49.262702 29529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:49.262974 29529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:49.264148 29529 hybrid_clock.cc:648] HybridClock initialized: now 1790462629264146 us; error 43 us; skew 500 ppm
I20260926 22:43:49.265405 29529 webserver.cc:460] Webserver started at http://127.11.107.195:33587/ using document root <none> and password file <none>
I20260926 22:43:49.265693 29529 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:49.265767 29529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:49.265941 29529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:49.266994 29529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "055004ba29ed43318985f20e3d35a7cc"
format_stamp: "Formatted at 2026-09-26 22:43:49 on dist-test-slave-j8sq"
server_key: "660008b4cfa795b6111d887e121784b1"
server_key_iv: "c683cd6fbb15248311849d4a388de191"
server_key_version: "encryptionkey@0"
I20260926 22:43:49.267347 29529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "055004ba29ed43318985f20e3d35a7cc"
format_stamp: "Formatted at 2026-09-26 22:43:49 on dist-test-slave-j8sq"
server_key: "660008b4cfa795b6111d887e121784b1"
server_key_iv: "c683cd6fbb15248311849d4a388de191"
server_key_version: "encryptionkey@0"
I20260926 22:43:49.268563 29529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:43:49.269222 29543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:49.269387 29529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:43:49.269480 29529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "055004ba29ed43318985f20e3d35a7cc"
format_stamp: "Formatted at 2026-09-26 22:43:49 on dist-test-slave-j8sq"
server_key: "660008b4cfa795b6111d887e121784b1"
server_key_iv: "c683cd6fbb15248311849d4a388de191"
server_key_version: "encryptionkey@0"
I20260926 22:43:49.269562 29529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:49.292441 29529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:49.292760 29529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:49.292915 29529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:49.293141 29529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:49.293476 29529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:49.293531 29529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:49.293565 29529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:49.293601 29529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:49.300041 29529 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:35371
I20260926 22:43:49.300079 29656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:35371 every 8 connection(s)
I20260926 22:43:49.300441 29529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:43:49.305591 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.301218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:46873 (local address 127.11.107.254:39603)
0926 22:43:49.301294 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:49.301296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:49.301563 (+ 267us) server_negotiation.cc:407] Connection header received
0926 22:43:49.302052 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:49.302053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:49.302064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:49.302128 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:49.302801 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.303301 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.304241 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.304419 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.304469 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.305338 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:49.305339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:49.305339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:49.305344 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:49.305359 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:49.305363 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:49.305419 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:49.305500 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 22:43:49.305525 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 22:43:49.305790 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.301212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:39603 (local address 127.11.107.195:46873)
0926 22:43:49.301428 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:49.301464 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:49.301956 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:49.302207 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:49.302210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:49.302218 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:49.302715 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.302719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.303386 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.303386 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.304170 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.304172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.304491 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.304492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.304596 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.305203 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:43:49.305203 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:49.305232 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:43:49.305254 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:49.305429 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:49.305433 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:49.305435 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:43:49.305486 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:43:49.305562 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
I20260926 22:43:49.306250 29657 heartbeater.cc:347] Connected to a master server at 127.11.107.254:39603
I20260926 22:43:49.306324 29657 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:49.306452 29657 heartbeater.cc:511] Master 127.11.107.254:39603 requested a full tablet report, sending...
I20260926 22:43:49.306793 29207 ts_manager.cc:194] Registered new tserver with Master: 055004ba29ed43318985f20e3d35a7cc (127.11.107.195:35371)
I20260926 22:43:49.307176 29207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.107.195:46873
I20260926 22:43:49.308300 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 29529
I20260926 22:43:49.308382 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:43:49.308521 11695 external_mini_cluster.cc:1428] Setting key 4c2a229ee58dbf9c3b37a254383dae9b
I20260926 22:43:49.309461 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:43:49.313458 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.310815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50492 (local address 127.11.107.254:39603)
0926 22:43:49.310886 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:49.310888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:49.311010 (+ 122us) server_negotiation.cc:407] Connection header received
0926 22:43:49.311034 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:49.311035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:49.311044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:49.311097 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:49.311296 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.311724 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.312444 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.312617 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.312668 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.313280 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:49.313281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:49.313281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:49.313285 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:49.313295 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:49.313298 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:49.313340 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:49.313392 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 22:43:49.313410 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 22:43:49.315071 29207 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:43:49.316591 29207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:43:49.322971 29666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.319067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:36895 (local address 127.0.0.1:33028)
0926 22:43:49.319465 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:49.319475 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:49.319522 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:49.319733 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:49.319734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:49.319736 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:49.319779 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.319781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.320708 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.320709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.321975 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.321978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.322694 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.322695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.322770 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.322784 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 22:43:49.322854 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 22:43:49.322900 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":4,"thread_start_us":74,"threads_started":1}
I20260926 22:43:49.322984 29670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.319168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.11.107.193:36895)
0926 22:43:49.319369 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:49.319373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:49.319579 (+ 206us) server_negotiation.cc:407] Connection header received
0926 22:43:49.319623 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:49.319626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:49.319641 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:49.319716 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:49.319847 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.320608 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.322058 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.322598 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.322684 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.322773 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:49.322868 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 22:43:49.322909 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":96,"threads_started":1}
I20260926 22:43:49.323313 29665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.318871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35503 (local address 127.0.0.1:32874)
0926 22:43:49.319208 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:49.319248 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:49.319319 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:49.319808 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:49.319809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:49.319811 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:49.319846 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.319848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.321033 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.321034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.322074 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.322076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.323033 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.323034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.323131 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.323137 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 22:43:49.323196 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 22:43:49.323235 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":7,"thread_start_us":75,"threads_started":1}
I20260926 22:43:49.323359 29668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.318951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.11.107.194:35503)
0926 22:43:49.319144 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:49.319147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:49.319309 (+ 162us) server_negotiation.cc:407] Connection header received
0926 22:43:49.319418 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:49.319420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:49.319437 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:49.319492 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:49.319907 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.320972 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.322159 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.322845 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.322918 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.322994 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:49.323225 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 22:43:49.323268 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
I20260926 22:43:49.323414 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.318931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.11.107.195:35371)
0926 22:43:49.319074 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:49.319093 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:49.319405 (+ 312us) server_negotiation.cc:407] Connection header received
0926 22:43:49.319447 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:49.319449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:49.319464 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:49.319537 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:49.319769 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.320865 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.322134 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.322845 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.322918 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.322994 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:49.323275 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 22:43:49.323312 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260926 22:43:49.323558 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.318867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:35371 (local address 127.0.0.1:40730)
0926 22:43:49.319208 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:49.319248 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:49.319319 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:49.319644 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:49.319647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:49.319648 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:49.319710 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.319712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.320957 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.320958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.322026 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.322028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.323035 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.323036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.323131 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.323139 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 22:43:49.323190 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:43:49.323237 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":31,"thread_start_us":56,"threads_started":1}
I20260926 22:43:49.323962 29329 tablet_service.cc:1467] Processing CreateTablet for tablet 406eb62f8ab64996982a0705aeabd642 (DEFAULT_TABLE table=test-table [id=80162fd4000348fc8b4be4da364e2eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:49.324023 29460 tablet_service.cc:1467] Processing CreateTablet for tablet 406eb62f8ab64996982a0705aeabd642 (DEFAULT_TABLE table=test-table [id=80162fd4000348fc8b4be4da364e2eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:49.324096 29591 tablet_service.cc:1467] Processing CreateTablet for tablet 406eb62f8ab64996982a0705aeabd642 (DEFAULT_TABLE table=test-table [id=80162fd4000348fc8b4be4da364e2eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:49.324281 29460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406eb62f8ab64996982a0705aeabd642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:49.324307 29329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406eb62f8ab64996982a0705aeabd642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:49.324363 29591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406eb62f8ab64996982a0705aeabd642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:49.326742 29671 tablet_bootstrap.cc:492] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91: Bootstrap starting.
I20260926 22:43:49.327438 29671 tablet_bootstrap.cc:654] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:49.327754 29671 log.cc:824] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:49.327752 29673 tablet_bootstrap.cc:492] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196: Bootstrap starting.
I20260926 22:43:49.327847 29672 tablet_bootstrap.cc:492] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc: Bootstrap starting.
I20260926 22:43:49.328392 29672 tablet_bootstrap.cc:654] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:49.328392 29673 tablet_bootstrap.cc:654] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:49.328716 29673 log.cc:824] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:49.328716 29672 log.cc:824] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:49.329455 29673 tablet_bootstrap.cc:492] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196: No bootstrap required, opened a new log
I20260926 22:43:49.329499 29671 tablet_bootstrap.cc:492] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91: No bootstrap required, opened a new log
I20260926 22:43:49.329522 29673 ts_tablet_manager.cc:1397] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:43:49.329566 29671 ts_tablet_manager.cc:1397] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 22:43:49.329704 29672 tablet_bootstrap.cc:492] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc: No bootstrap required, opened a new log
I20260926 22:43:49.329759 29672 ts_tablet_manager.cc:1397] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:43:49.331410 29671 raft_consensus.cc:348] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055004ba29ed43318985f20e3d35a7cc" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35371 } } peers { permanent_uuid: "08b3c05c22394f8a8e555dcc640a3e91" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35503 } } peers { permanent_uuid: "33051d5add924b6dbdffc84f473be196" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36895 } }
I20260926 22:43:49.331561 29671 raft_consensus.cc:374] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:49.331605 29671 raft_consensus.cc:729] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b3c05c22394f8a8e555dcc640a3e91, State: Initialized, Role: FOLLOWER
I20260926 22:43:49.331710 29671 consensus_queue.cc:260] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055004ba29ed43318985f20e3d35a7cc" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35371 } } peers { permanent_uuid: "08b3c05c22394f8a8e555dcc640a3e91" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35503 } } peers { permanent_uuid: "33051d5add924b6dbdffc84f473be196" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36895 } }
I20260926 22:43:49.331965 29671 ts_tablet_manager.cc:1428] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:43:49.332104 29526 heartbeater.cc:503] Master 127.11.107.254:39603 was elected leader, sending a full tablet report...
I20260926 22:43:49.332151 29673 raft_consensus.cc:348] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055004ba29ed43318985f20e3d35a7cc" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35371 } } peers { permanent_uuid: "08b3c05c22394f8a8e555dcc640a3e91" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35503 } } peers { permanent_uuid: "33051d5add924b6dbdffc84f473be196" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36895 } }
I20260926 22:43:49.332206 29672 raft_consensus.cc:348] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055004ba29ed43318985f20e3d35a7cc" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35371 } } peers { permanent_uuid: "08b3c05c22394f8a8e555dcc640a3e91" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35503 } } peers { permanent_uuid: "33051d5add924b6dbdffc84f473be196" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36895 } }
I20260926 22:43:49.332324 29673 raft_consensus.cc:374] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:49.332340 29672 raft_consensus.cc:374] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:49.332371 29672 raft_consensus.cc:729] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055004ba29ed43318985f20e3d35a7cc, State: Initialized, Role: FOLLOWER
I20260926 22:43:49.332371 29673 raft_consensus.cc:729] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33051d5add924b6dbdffc84f473be196, State: Initialized, Role: FOLLOWER
I20260926 22:43:49.332484 29673 consensus_queue.cc:260] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055004ba29ed43318985f20e3d35a7cc" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35371 } } peers { permanent_uuid: "08b3c05c22394f8a8e555dcc640a3e91" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35503 } } peers { permanent_uuid: "33051d5add924b6dbdffc84f473be196" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36895 } }
I20260926 22:43:49.332495 29672 consensus_queue.cc:260] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055004ba29ed43318985f20e3d35a7cc" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35371 } } peers { permanent_uuid: "08b3c05c22394f8a8e555dcc640a3e91" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35503 } } peers { permanent_uuid: "33051d5add924b6dbdffc84f473be196" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36895 } }
I20260926 22:43:49.332733 29673 ts_tablet_manager.cc:1428] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 22:43:49.332762 29657 heartbeater.cc:503] Master 127.11.107.254:39603 was elected leader, sending a full tablet report...
I20260926 22:43:49.332842 29395 heartbeater.cc:503] Master 127.11.107.254:39603 was elected leader, sending a full tablet report...
I20260926 22:43:49.332733 29672 ts_tablet_manager.cc:1428] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:43:49.409031 29679 raft_consensus.cc:482] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:49.409166 29679 raft_consensus.cc:504] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055004ba29ed43318985f20e3d35a7cc" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35371 } } peers { permanent_uuid: "08b3c05c22394f8a8e555dcc640a3e91" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35503 } } peers { permanent_uuid: "33051d5add924b6dbdffc84f473be196" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36895 } }
I20260926 22:43:49.409678 29679 leader_election.cc:288] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08b3c05c22394f8a8e555dcc640a3e91 (127.11.107.194:35503), 33051d5add924b6dbdffc84f473be196 (127.11.107.193:36895)
W20260926 22:43:49.413280 29527 tablet.cc:2369] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:43:49.413582 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.409885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:36895 (local address 127.11.107.195:38025)
0926 22:43:49.409943 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:49.409955 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:49.410004 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:49.410170 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:49.410171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:49.410173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:49.410256 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.410259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.411209 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.411210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.412489 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.412492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.413283 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.413284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.413368 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.413372 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:43:49.413436 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 22:43:49.413499 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":67}
I20260926 22:43:49.413616 29670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.409952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:38025 (local address 127.11.107.193:36895)
0926 22:43:49.410015 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:49.410017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:49.410032 (+ 15us) server_negotiation.cc:407] Connection header received
0926 22:43:49.410081 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:49.410082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:49.410096 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:49.410161 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:49.410336 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.411138 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.412654 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.413174 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.413251 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.413259 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:49.413507 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 22:43:49.413550 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 22:43:49.413616 29668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.409977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:48319 (local address 127.11.107.194:35503)
0926 22:43:49.410024 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:49.410026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:49.410179 (+ 153us) server_negotiation.cc:407] Connection header received
0926 22:43:49.410214 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:49.410215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:49.410227 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:49.410288 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:49.410483 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.411011 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.412605 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.413174 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.413251 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.413259 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:49.413507 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 22:43:49.413550 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:43:49.413885 29680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.409915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:35503 (local address 127.11.107.195:48319)
0926 22:43:49.410068 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:49.410077 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:49.410137 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:49.410345 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:49.410345 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:49.410347 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:49.410410 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.410412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.411129 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.411130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.412505 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:49.412507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.413320 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.413321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:49.413420 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.413424 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:43:49.413718 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 22:43:49.413773 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":32,"thread_start_us":89,"threads_started":1}
I20260926 22:43:49.414008 29349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406eb62f8ab64996982a0705aeabd642" candidate_uuid: "055004ba29ed43318985f20e3d35a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33051d5add924b6dbdffc84f473be196" is_pre_election: true
I20260926 22:43:49.414167 29349 raft_consensus.cc:2454] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 055004ba29ed43318985f20e3d35a7cc in term 0.
I20260926 22:43:49.414160 29480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406eb62f8ab64996982a0705aeabd642" candidate_uuid: "055004ba29ed43318985f20e3d35a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b3c05c22394f8a8e555dcc640a3e91" is_pre_election: true
I20260926 22:43:49.414275 29480 raft_consensus.cc:2454] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 055004ba29ed43318985f20e3d35a7cc in term 0.
I20260926 22:43:49.414407 29545 leader_election.cc:302] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055004ba29ed43318985f20e3d35a7cc, 33051d5add924b6dbdffc84f473be196; no voters:
I20260926 22:43:49.414539 29679 raft_consensus.cc:2781] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:43:49.414609 29679 raft_consensus.cc:482] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:43:49.414644 29679 raft_consensus.cc:3037] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:49.415355 29679 raft_consensus.cc:504] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055004ba29ed43318985f20e3d35a7cc" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35371 } } peers { permanent_uuid: "08b3c05c22394f8a8e555dcc640a3e91" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35503 } } peers { permanent_uuid: "33051d5add924b6dbdffc84f473be196" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36895 } }
I20260926 22:43:49.415500 29679 leader_election.cc:288] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [CANDIDATE]: Term 1 election: Requested vote from peers 08b3c05c22394f8a8e555dcc640a3e91 (127.11.107.194:35503), 33051d5add924b6dbdffc84f473be196 (127.11.107.193:36895)
I20260926 22:43:49.415663 29349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406eb62f8ab64996982a0705aeabd642" candidate_uuid: "055004ba29ed43318985f20e3d35a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33051d5add924b6dbdffc84f473be196"
I20260926 22:43:49.415678 29480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406eb62f8ab64996982a0705aeabd642" candidate_uuid: "055004ba29ed43318985f20e3d35a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b3c05c22394f8a8e555dcc640a3e91"
I20260926 22:43:49.415743 29349 raft_consensus.cc:3037] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:49.415773 29480 raft_consensus.cc:3037] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:49.416436 29349 raft_consensus.cc:2454] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 055004ba29ed43318985f20e3d35a7cc in term 1.
I20260926 22:43:49.416468 29480 raft_consensus.cc:2454] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 055004ba29ed43318985f20e3d35a7cc in term 1.
I20260926 22:43:49.416641 29545 leader_election.cc:302] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055004ba29ed43318985f20e3d35a7cc, 33051d5add924b6dbdffc84f473be196; no voters:
I20260926 22:43:49.416749 29679 raft_consensus.cc:2781] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:49.416877 29679 raft_consensus.cc:686] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [term 1 LEADER]: Becoming Leader. State: Replica: 055004ba29ed43318985f20e3d35a7cc, State: Running, Role: LEADER
I20260926 22:43:49.417070 29679 consensus_queue.cc:237] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055004ba29ed43318985f20e3d35a7cc" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35371 } } peers { permanent_uuid: "08b3c05c22394f8a8e555dcc640a3e91" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35503 } } peers { permanent_uuid: "33051d5add924b6dbdffc84f473be196" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36895 } }
I20260926 22:43:49.417932 29207 catalog_manager.cc:5495] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 055004ba29ed43318985f20e3d35a7cc (127.11.107.195). New cstate: current_term: 1 leader_uuid: "055004ba29ed43318985f20e3d35a7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055004ba29ed43318985f20e3d35a7cc" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 35371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08b3c05c22394f8a8e555dcc640a3e91" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33051d5add924b6dbdffc84f473be196" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36895 } health_report { overall_health: UNKNOWN } } }
I20260926 22:43:49.440204 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.437782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40736 (local address 127.11.107.195:35371)
0926 22:43:49.437861 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:49.437863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:49.437872 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:43:49.437915 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:49.437915 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:49.437932 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:49.438010 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:43:49.438155 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.438606 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.439369 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:49.439567 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:49.439699 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:49.439874 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:43:49.439977 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:43:49.440070 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:49.440087 (+ 17us) server_negotiation.cc:299] Negotiation successful
0926 22:43:49.440105 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 22:43:49.442090 29349 raft_consensus.cc:1261] T 406eb62f8ab64996982a0705aeabd642 P 33051d5add924b6dbdffc84f473be196 [term 1 FOLLOWER]: Refusing update from remote peer 055004ba29ed43318985f20e3d35a7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:49.442112 29480 raft_consensus.cc:1261] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91 [term 1 FOLLOWER]: Refusing update from remote peer 055004ba29ed43318985f20e3d35a7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:49.442447 29681 consensus_queue.cc:1035] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b3c05c22394f8a8e555dcc640a3e91" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 35503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:49.442610 29679 consensus_queue.cc:1035] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33051d5add924b6dbdffc84f473be196" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:49.445307 29685 mvcc.cc:205] Tried to move back new op lower bound from 7333734930191495168 to 7333734930093174784. Current Snapshot: MvccSnapshot[applied={T|T < 7333734930191495168}]
I20260926 22:43:49.446743 29686 mvcc.cc:205] Tried to move back new op lower bound from 7333734930191495168 to 7333734930093174784. Current Snapshot: MvccSnapshot[applied={T|T < 7333734930191495168 or (T in {7333734930191495168})}]
I20260926 22:43:49.446861 29205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:49.446944 29205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:49.447654 29205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bed74152d694a35a771f8b45ff5f0d5: Sending DeleteTablet for 3 replicas of tablet 406eb62f8ab64996982a0705aeabd642
I20260926 22:43:49.447956 29460 tablet_service.cc:1514] Processing DeleteTablet for tablet 406eb62f8ab64996982a0705aeabd642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:49 UTC) from {username='slave'} at 127.0.0.1:32874
I20260926 22:43:49.448005 29591 tablet_service.cc:1514] Processing DeleteTablet for tablet 406eb62f8ab64996982a0705aeabd642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:49 UTC) from {username='slave'} at 127.0.0.1:40730
I20260926 22:43:49.448020 29329 tablet_service.cc:1514] Processing DeleteTablet for tablet 406eb62f8ab64996982a0705aeabd642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:49 UTC) from {username='slave'} at 127.0.0.1:33028
I20260926 22:43:49.448320 29695 tablet_replica.cc:330] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc: stopping tablet replica
I20260926 22:43:49.448416 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 29267
I20260926 22:43:49.448441 29695 raft_consensus.cc:2229] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:43:49.448534 29695 raft_consensus.cc:2258] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:49.448673 29694 tablet_replica.cc:330] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91: stopping tablet replica
I20260926 22:43:49.448747 29694 raft_consensus.cc:2229] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:43:49.448832 29694 raft_consensus.cc:2258] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:49.449421 29695 ts_tablet_manager.cc:1905] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:49.450172 29694 ts_tablet_manager.cc:1905] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:49.451431 29695 ts_tablet_manager.cc:1918] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:49.451504 29695 log.cc:1196] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/406eb62f8ab64996982a0705aeabd642
I20260926 22:43:49.451764 29695 ts_tablet_manager.cc:1939] T 406eb62f8ab64996982a0705aeabd642 P 055004ba29ed43318985f20e3d35a7cc: Deleting consensus metadata
I20260926 22:43:49.452006 29694 ts_tablet_manager.cc:1918] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:49.452209 29694 log.cc:1196] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/406eb62f8ab64996982a0705aeabd642
I20260926 22:43:49.452184 29192 catalog_manager.cc:4841] TS 055004ba29ed43318985f20e3d35a7cc (127.11.107.195:35371): tablet 406eb62f8ab64996982a0705aeabd642 (table test-table [id=80162fd4000348fc8b4be4da364e2eec]) successfully deleted
I20260926 22:43:49.452557 29694 ts_tablet_manager.cc:1939] T 406eb62f8ab64996982a0705aeabd642 P 08b3c05c22394f8a8e555dcc640a3e91: Deleting consensus metadata
I20260926 22:43:49.453020 29194 catalog_manager.cc:4841] TS 08b3c05c22394f8a8e555dcc640a3e91 (127.11.107.194:35503): tablet 406eb62f8ab64996982a0705aeabd642 (table test-table [id=80162fd4000348fc8b4be4da364e2eec]) successfully deleted
W20260926 22:43:49.455219 29192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:36895 (error 108)
I20260926 22:43:49.455683 29666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:49.455546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:36895 (local address 127.0.0.1:33040)
0926 22:43:49.455603 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:49.455650 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:36895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 22:43:49.455762 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 29398
W20260926 22:43:49.456918 29192 catalog_manager.cc:4584] TS 33051d5add924b6dbdffc84f473be196 (127.11.107.193:36895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 406eb62f8ab64996982a0705aeabd642: Network error: Client connection negotiation failed: client connection to 127.11.107.193:36895: connect: Connection refused (error 111)
I20260926 22:43:49.460852 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 29529
I20260926 22:43:49.465610 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 29176
2026-09-26T22:43:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (732 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:43:49 dist-test-slave-j8sq krb5kdc[29700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:43:49 dist-test-slave-j8sq krb5kdc[29700](info): set up 2 sockets
Sep 26 22:43:49 dist-test-slave-j8sq krb5kdc[29700](info): commencing operation
krb5kdc: starting...
W20260926 22:43:50.498693 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:43:50 dist-test-slave-j8sq krb5kdc[29700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:43:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:43:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:43:50.616427 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:33775
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:33775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:50.695606 29713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:50.695833 29713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:50.695870 29713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:50.697024 29713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:43:50.697093 29713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:50.697115 29713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:50.697137 29713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:43:50.697166 29713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:43:50.698616 29713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:33775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:33775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:50.698889 29713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:50.699115 29713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:50.703872 29718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:50.703941 29721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:50.703924 29719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:50.704288 29713 server_base.cc:1034] running on GCE node
I20260926 22:43:50.704557 29713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:50.704826 29713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:50.706003 29713 hybrid_clock.cc:648] HybridClock initialized: now 1790462630705987 us; error 41 us; skew 500 ppm
Sep 26 22:43:50 dist-test-slave-j8sq krb5kdc[29700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462630, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:50.708597 29713 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:43:50.709043 29713 webserver.cc:460] Webserver started at http://127.11.107.254:43059/ using document root <none> and password file <none>
I20260926 22:43:50.709242 29713 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:50.709285 29713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:50.709381 29713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:50.710541 29713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "77383cece6f7429993fb77b2d62f5c5c"
format_stamp: "Formatted at 2026-09-26 22:43:50 on dist-test-slave-j8sq"
server_key: "091d2a70fce03762805435fa1be26c74"
server_key_iv: "aacaa227b44fdc02ef02e57ced9f97db"
server_key_version: "encryptionkey@0"
I20260926 22:43:50.710903 29713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "77383cece6f7429993fb77b2d62f5c5c"
format_stamp: "Formatted at 2026-09-26 22:43:50 on dist-test-slave-j8sq"
server_key: "091d2a70fce03762805435fa1be26c74"
server_key_iv: "aacaa227b44fdc02ef02e57ced9f97db"
server_key_version: "encryptionkey@0"
I20260926 22:43:50.712256 29713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:43:50.712939 29728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:50.713110 29713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:43:50.713196 29713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "77383cece6f7429993fb77b2d62f5c5c"
format_stamp: "Formatted at 2026-09-26 22:43:50 on dist-test-slave-j8sq"
server_key: "091d2a70fce03762805435fa1be26c74"
server_key_iv: "aacaa227b44fdc02ef02e57ced9f97db"
server_key_version: "encryptionkey@0"
I20260926 22:43:50.713249 29713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:50.728425 29713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:50.730705 29713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:50.730908 29713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:50.734750 29713 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:33775
I20260926 22:43:50.734808 29780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:33775 every 8 connection(s)
I20260926 22:43:50.735129 29713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:43:50.736074 29781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:50.738302 29781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c: Bootstrap starting.
I20260926 22:43:50.738811 29781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:50.739034 29781 log.cc:824] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:50.739694 29781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c: No bootstrap required, opened a new log
I20260926 22:43:50.741101 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 29713
I20260926 22:43:50.741212 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:43:50.741403 11695 external_mini_cluster.cc:1428] Setting key 2337005ad6ca1d48aa7e1fd031c8465e
I20260926 22:43:50.741464 29781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77383cece6f7429993fb77b2d62f5c5c" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 33775 } }
I20260926 22:43:50.741662 29781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:50.741709 29781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77383cece6f7429993fb77b2d62f5c5c, State: Initialized, Role: FOLLOWER
I20260926 22:43:50.742136 29781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77383cece6f7429993fb77b2d62f5c5c" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 33775 } }
I20260926 22:43:50.742235 29781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:43:50.742260 29781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:43:50.742328 29781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:50.743448 29781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77383cece6f7429993fb77b2d62f5c5c" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 33775 } }
I20260926 22:43:50.743533 29781 leader_election.cc:302] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77383cece6f7429993fb77b2d62f5c5c; no voters:
I20260926 22:43:50.743713 29781 leader_election.cc:288] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:43:50.743744 29786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:50.743848 29786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [term 1 LEADER]: Becoming Leader. State: Replica: 77383cece6f7429993fb77b2d62f5c5c, State: Running, Role: LEADER
I20260926 22:43:50.743959 29781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:43:50.743937 29786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77383cece6f7429993fb77b2d62f5c5c" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 33775 } }
I20260926 22:43:50.744300 29786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77383cece6f7429993fb77b2d62f5c5c. Latest consensus state: current_term: 1 leader_uuid: "77383cece6f7429993fb77b2d62f5c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77383cece6f7429993fb77b2d62f5c5c" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 33775 } } }
I20260926 22:43:50.744391 29786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:50.744310 29788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77383cece6f7429993fb77b2d62f5c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77383cece6f7429993fb77b2d62f5c5c" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 33775 } } }
I20260926 22:43:50.744553 29788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:50.744635 29792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:43:50.745143 29792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:43:50.747527 29792 catalog_manager.cc:1347] Generated new cluster ID: 1c7ffc307d9b4384ab8c11bfc04aa2df
I20260926 22:43:50.747584 29792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:43:50 dist-test-slave-j8sq krb5kdc[29700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:50.751979 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:50.741932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.11.107.254:33775)
0926 22:43:50.742122 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:50.742130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:50.742155 (+ 25us) server_negotiation.cc:407] Connection header received
0926 22:43:50.742269 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:50.742277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:50.742356 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:50.742502 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:50.743304 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:50.744546 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:50.745656 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:50.746264 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:50.746382 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:50.748792 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:50.748796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:50.748796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:50.748808 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:50.750496 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:50.750864 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:50.750867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:50.750867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:50.750966 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:50.751183 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:50.751184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:50.751185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:50.751409 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:50.751586 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:50.751652 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 22:43:50.751791 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260926 22:43:50.771247 29792 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:43:50.771961 29792 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:43:50.775612 29792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c: Generated new TSK 0
I20260926 22:43:50.775786 29792 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:43:50.808456 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:33775
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:50.888279 29807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:50.888504 29807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:50.888540 29807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:50.889685 29807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:50.889750 29807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:50.889794 29807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:43:50.891239 29807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:33775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:50.891486 29807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:50.891707 29807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:43:50.896735 29807 server_base.cc:1034] running on GCE node
W20260926 22:43:50.896713 29812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:50.896719 29813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:50.896731 29815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:50.897089 29807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:50.897331 29807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:50.898488 29807 hybrid_clock.cc:648] HybridClock initialized: now 1790462630898466 us; error 40 us; skew 500 ppm
Sep 26 22:43:50 dist-test-slave-j8sq krb5kdc[29700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462630, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:50.900918 29807 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:43:50.901381 29807 webserver.cc:460] Webserver started at http://127.11.107.193:42555/ using document root <none> and password file <none>
I20260926 22:43:50.901598 29807 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:50.901672 29807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:50.901820 29807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:50.902863 29807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "49b48c6fcf7c4dbd84af0c4139534b2a"
format_stamp: "Formatted at 2026-09-26 22:43:50 on dist-test-slave-j8sq"
server_key: "1d29fec83b5ffac71de7ed0cdc6779e3"
server_key_iv: "b422f1e713e2db4cf8d35b5d9c4404a2"
server_key_version: "encryptionkey@0"
I20260926 22:43:50.903213 29807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "49b48c6fcf7c4dbd84af0c4139534b2a"
format_stamp: "Formatted at 2026-09-26 22:43:50 on dist-test-slave-j8sq"
server_key: "1d29fec83b5ffac71de7ed0cdc6779e3"
server_key_iv: "b422f1e713e2db4cf8d35b5d9c4404a2"
server_key_version: "encryptionkey@0"
I20260926 22:43:50.904512 29807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:43:50.905228 29822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:50.905402 29807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:43:50.905499 29807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "49b48c6fcf7c4dbd84af0c4139534b2a"
format_stamp: "Formatted at 2026-09-26 22:43:50 on dist-test-slave-j8sq"
server_key: "1d29fec83b5ffac71de7ed0cdc6779e3"
server_key_iv: "b422f1e713e2db4cf8d35b5d9c4404a2"
server_key_version: "encryptionkey@0"
I20260926 22:43:50.905581 29807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:50.922780 29807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:50.924569 29807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:50.924737 29807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:50.924966 29807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:50.925276 29807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:50.925331 29807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:50.925371 29807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:50.925400 29807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:50.931191 29807 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:37185
I20260926 22:43:50.931252 29935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:37185 every 8 connection(s)
I20260926 22:43:50.931542 29807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:43:50.933275 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 29807
I20260926 22:43:50.933408 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:43:50.933612 11695 external_mini_cluster.cc:1428] Setting key 3703d4e21175d0ed37cdc726f64d53c9
Sep 26 22:43:50 dist-test-slave-j8sq krb5kdc[29700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462630, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:50.941001 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:50.932169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:37013 (local address 127.11.107.254:33775)
0926 22:43:50.932237 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:50.932239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:50.932434 (+ 195us) server_negotiation.cc:407] Connection header received
0926 22:43:50.932915 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:50.932916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:50.932930 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:50.932977 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:50.934071 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:50.934561 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:50.935569 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:50.935776 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:50.935865 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:50.938235 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:50.938236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:50.938237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:50.938244 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:50.939614 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:50.940069 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:50.940070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:50.940070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:50.940129 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:50.940383 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:50.940383 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:50.940384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:50.940494 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:50.940567 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:50.940830 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 22:43:50.940909 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
I20260926 22:43:50.941223 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:50.932160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:33775 (local address 127.11.107.193:37013)
0926 22:43:50.932318 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:50.932352 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:50.932831 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:50.933034 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:50.933038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:50.933698 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:50.933981 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:50.933990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:50.934673 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:50.934674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:50.935462 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:50.935464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:50.935888 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:50.935889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:50.936022 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:50.936522 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:50.936530 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:50.938142 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:50.939757 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:50.939764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:50.939765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:50.939979 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:50.940206 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:50.940207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:50.940207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:50.940311 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:50.940559 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:50.940566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:50.940700 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 22:43:50.940807 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 22:43:50.940958 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260926 22:43:50.941768 29936 heartbeater.cc:347] Connected to a master server at 127.11.107.254:33775
I20260926 22:43:50.941900 29936 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:50.942077 29936 heartbeater.cc:511] Master 127.11.107.254:33775 requested a full tablet report, sending...
I20260926 22:43:50.942610 29745 ts_manager.cc:194] Registered new tserver with Master: 49b48c6fcf7c4dbd84af0c4139534b2a (127.11.107.193:37185)
I20260926 22:43:50.943199 29745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:37013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:43:50.958459 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:33775
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:51.038190 29941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:51.038389 29941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:51.038414 29941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:51.039496 29941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:51.039530 29941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:51.039556 29941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:43:51.040917 29941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:33775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:51.041098 29941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:51.041268 29941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:51.046563 29949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:51.046573 29946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:51.046653 29947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:51.046816 29941 server_base.cc:1034] running on GCE node
I20260926 22:43:51.047036 29941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:51.047279 29941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:51.048424 29941 hybrid_clock.cc:648] HybridClock initialized: now 1790462631048408 us; error 34 us; skew 500 ppm
Sep 26 22:43:51 dist-test-slave-j8sq krb5kdc[29700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462631, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:51.050943 29941 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:43:51.051411 29941 webserver.cc:460] Webserver started at http://127.11.107.194:37459/ using document root <none> and password file <none>
I20260926 22:43:51.051649 29941 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:51.051717 29941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:51.051872 29941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:51.052933 29941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "5f530f831e914ab4a0adb4ac264e5704"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "94e7bfc937bfce72cf43b2a0855784fd"
server_key_iv: "159be6f35add33c76478ff75f92c4035"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.053299 29941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "5f530f831e914ab4a0adb4ac264e5704"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "94e7bfc937bfce72cf43b2a0855784fd"
server_key_iv: "159be6f35add33c76478ff75f92c4035"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.054733 29941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:43:51.055544 29956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:51.055759 29941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 22:43:51.055873 29941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "5f530f831e914ab4a0adb4ac264e5704"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "94e7bfc937bfce72cf43b2a0855784fd"
server_key_iv: "159be6f35add33c76478ff75f92c4035"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.055949 29941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:51.066848 29941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:51.068922 29941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:51.069087 29941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:51.069311 29941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:51.069615 29941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:51.069667 29941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:51.069707 29941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:51.069736 29941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:51.075774 29941 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:33927
I20260926 22:43:51.075822 30069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:33927 every 8 connection(s)
I20260926 22:43:51.076130 29941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:43:51 dist-test-slave-j8sq krb5kdc[29700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462631, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:51.083011 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 29941
I20260926 22:43:51.083137 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:43:51.083365 11695 external_mini_cluster.cc:1428] Setting key becd95e31d95e458e569988aaf7daed7
I20260926 22:43:51.086127 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.076851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:38163 (local address 127.11.107.254:33775)
0926 22:43:51.076907 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:51.076909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:51.077159 (+ 250us) server_negotiation.cc:407] Connection header received
0926 22:43:51.077687 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:51.077688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:51.077701 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:51.077754 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:51.078841 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.079413 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.080432 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.080632 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.080705 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.082822 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:51.082823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:51.082823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:51.082830 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:51.084489 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:51.084991 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:51.084991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:51.084992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:51.085050 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:51.085332 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:51.085332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:51.085332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:51.085467 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:51.085548 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:51.085897 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 22:43:51.086003 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 22:43:51.086156 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.076837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:33775 (local address 127.11.107.194:38163)
0926 22:43:51.077023 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:51.077056 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:51.077568 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:51.077822 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:51.077826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:51.078500 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:51.078709 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.078715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.079515 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.079517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.080332 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.080334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.080722 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.080723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.080853 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.081357 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:51.081366 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:51.082734 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:51.084574 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:51.084581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:51.084584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:51.084911 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:51.085126 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:51.085127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:51.085127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:51.085267 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:51.085548 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:51.085555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:51.085718 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 22:43:51.085811 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 22:43:51.085999 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
I20260926 22:43:51.086879 30070 heartbeater.cc:347] Connected to a master server at 127.11.107.254:33775
I20260926 22:43:51.086994 30070 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:51.087162 30070 heartbeater.cc:511] Master 127.11.107.254:33775 requested a full tablet report, sending...
I20260926 22:43:51.087518 29745 ts_manager.cc:194] Registered new tserver with Master: 5f530f831e914ab4a0adb4ac264e5704 (127.11.107.194:33927)
I20260926 22:43:51.087970 29745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:38163
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:43:51.109642 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:33775
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:51.190938 30075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:51.191187 30075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:51.191227 30075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:51.192377 30075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:51.192448 30075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:51.192497 30075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:43:51.194007 30075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:33775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:51.194275 30075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:51.194505 30075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:51.199723 30083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:51.199798 30075 server_base.cc:1034] running on GCE node
W20260926 22:43:51.199724 30081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:51.199779 30080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:51.200171 30075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:51.200438 30075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:51.201611 30075 hybrid_clock.cc:648] HybridClock initialized: now 1790462631201590 us; error 34 us; skew 500 ppm
Sep 26 22:43:51 dist-test-slave-j8sq krb5kdc[29700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462631, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:51.204295 30075 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:43:51.204753 30075 webserver.cc:460] Webserver started at http://127.11.107.195:38553/ using document root <none> and password file <none>
I20260926 22:43:51.204981 30075 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:51.205052 30075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:51.205209 30075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:51.206322 30075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "e130bca81c1a46c18e85464127093a6b"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "8e40f7118b0712ff3dc67e38e95da1cd"
server_key_iv: "942459f253fb895efae783dc1473309b"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.206686 30075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "e130bca81c1a46c18e85464127093a6b"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "8e40f7118b0712ff3dc67e38e95da1cd"
server_key_iv: "942459f253fb895efae783dc1473309b"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.208045 30075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:51.208885 30090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:51.209066 30075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:43:51.209177 30075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "e130bca81c1a46c18e85464127093a6b"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "8e40f7118b0712ff3dc67e38e95da1cd"
server_key_iv: "942459f253fb895efae783dc1473309b"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.209261 30075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:51.216470 30075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:51.218632 30075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:51.218792 30075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:51.219010 30075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:51.219350 30075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:51.219405 30075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:51.219445 30075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:51.219473 30075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:51.225293 30075 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:34377
I20260926 22:43:51.225342 30203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:34377 every 8 connection(s)
I20260926 22:43:51.225641 30075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:43:51 dist-test-slave-j8sq krb5kdc[29700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462631, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:51.234417 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 30075
I20260926 22:43:51.234550 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:43:51.234760 11695 external_mini_cluster.cc:1428] Setting key a46add3ba12d38d517ec5412c3778be7
I20260926 22:43:51.235031 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.226362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:44603 (local address 127.11.107.254:33775)
0926 22:43:51.226442 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:51.226445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:51.226626 (+ 181us) server_negotiation.cc:407] Connection header received
0926 22:43:51.227109 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:51.227110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:51.227121 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:51.227195 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:51.228157 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.228641 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.229578 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.229778 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.229873 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.232228 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:51.232229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:51.232229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:51.232236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:51.233524 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:51.233953 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:51.233954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:51.233954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:51.234011 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:51.234268 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:51.234269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:51.234269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:51.234385 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:51.234488 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:51.234852 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 22:43:51.234940 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 22:43:51.235217 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.226329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:33775 (local address 127.11.107.195:44603)
0926 22:43:51.226482 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:51.226522 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:51.227018 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:51.227271 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:51.227274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:51.227882 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:51.228069 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.228074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.228724 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.228725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.229476 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.229478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.229905 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.229905 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.230025 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.230497 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:51.230506 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:51.232087 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:51.233598 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:51.233604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:51.233605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:51.233820 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:51.234074 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:51.234075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:51.234075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:51.234173 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:51.234480 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:51.234487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:51.234704 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 22:43:51.234826 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 22:43:51.234986 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260926 22:43:51.235860 30204 heartbeater.cc:347] Connected to a master server at 127.11.107.254:33775
I20260926 22:43:51.235977 30204 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:51.236169 30204 heartbeater.cc:511] Master 127.11.107.254:33775 requested a full tablet report, sending...
I20260926 22:43:51.236541 29745 ts_manager.cc:194] Registered new tserver with Master: e130bca81c1a46c18e85464127093a6b (127.11.107.195:34377)
I20260926 22:43:51.236913 29745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:44603
I20260926 22:43:51.246546 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:43:51.254158 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.247898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.11.107.254:33775)
0926 22:43:51.247974 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:51.247977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:51.248063 (+ 86us) server_negotiation.cc:407] Connection header received
0926 22:43:51.248140 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:51.248141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:51.248151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:51.248219 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:51.248789 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.249257 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.250062 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.250253 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.250326 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.251567 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:51.251568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:51.251569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:51.251576 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:51.252880 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:51.253276 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:51.253277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:51.253278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:51.253338 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:51.253559 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:51.253560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:51.253560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:51.253695 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:51.253817 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:51.253979 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 22:43:51.254041 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 22:43:51.255774 29745 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:43:51.257252 29745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:43:51.263827 30212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.259602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:34377 (local address 127.0.0.1:35748)
0926 22:43:51.259719 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:51.259730 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:51.259812 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:51.260126 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:51.260129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:51.260131 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:51.260198 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.260200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.261613 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.261614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.262635 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.262638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.263524 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.263525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.263611 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.263627 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 22:43:51.263672 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 22:43:51.263722 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":50,"threads_started":1}
I20260926 22:43:51.263949 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.259541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.11.107.194:33927)
0926 22:43:51.260009 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:51.260012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:51.260021 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:43:51.260051 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:51.260053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:51.260065 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:51.260169 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:51.260405 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.261233 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.262857 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.263557 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.263625 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.263682 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:51.263824 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 22:43:51.263877 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":88,"threads_started":1}
I20260926 22:43:51.264099 30217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.259814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.11.107.195:34377)
0926 22:43:51.259969 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:51.259974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:51.259982 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:43:51.260015 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:51.260017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:51.260030 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:51.260102 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:51.260369 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.261491 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.262713 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.263421 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.263896 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.263956 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:51.263981 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 22:43:51.264024 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260926 22:43:51.264159 30216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.259444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:33927 (local address 127.0.0.1:55812)
0926 22:43:51.259685 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:51.259697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:51.259748 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:51.260189 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:51.260190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:51.260192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:51.260267 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.260269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.261346 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.261347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.262764 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.262767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.263655 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.263656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.263731 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.263735 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:43:51.263793 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 22:43:51.263841 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":20,"thread_start_us":78,"threads_started":1}
I20260926 22:43:51.264552 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.259397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:37185 (local address 127.0.0.1:51126)
0926 22:43:51.259602 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:51.259649 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:51.259719 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:51.260509 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:51.260509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:51.260511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:51.260557 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.260558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.261763 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.261764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.262986 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.262988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.263825 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.263825 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.263884 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.263888 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:43:51.264435 (+ 547us) client_negotiation.cc:240] Negotiation successful
0926 22:43:51.264486 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":30,"thread_start_us":64,"threads_started":1}
I20260926 22:43:51.264886 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdff6d8698e4f3f9c5cae19b6c90f1c (DEFAULT_TABLE table=test-table [id=06455b255e2c483a898b3202e8eb7eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:51.264942 30004 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdff6d8698e4f3f9c5cae19b6c90f1c (DEFAULT_TABLE table=test-table [id=06455b255e2c483a898b3202e8eb7eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:51.265232 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdff6d8698e4f3f9c5cae19b6c90f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:51.265232 30004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdff6d8698e4f3f9c5cae19b6c90f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:51.268519 30218 tablet_bootstrap.cc:492] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b: Bootstrap starting.
I20260926 22:43:51.268617 30214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.259491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51126 (local address 127.11.107.193:37185)
0926 22:43:51.260298 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:51.260302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:51.260312 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:43:51.260349 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:51.260351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:51.260366 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:51.260438 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:51.260633 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.261661 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.263073 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.263652 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.263730 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.263790 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:51.264451 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 22:43:51.268535 (+ 4084us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":86,"threads_started":1}
I20260926 22:43:51.269171 30218 tablet_bootstrap.cc:654] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:51.269157 29870 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdff6d8698e4f3f9c5cae19b6c90f1c (DEFAULT_TABLE table=test-table [id=06455b255e2c483a898b3202e8eb7eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:51.269482 29870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdff6d8698e4f3f9c5cae19b6c90f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:51.269724 30218 log.cc:824] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:51.269989 30219 tablet_bootstrap.cc:492] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704: Bootstrap starting.
I20260926 22:43:51.270589 30218 tablet_bootstrap.cc:492] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b: No bootstrap required, opened a new log
I20260926 22:43:51.270665 30219 tablet_bootstrap.cc:654] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:51.270669 30218 ts_tablet_manager.cc:1397] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:51.271238 30219 log.cc:824] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:51.271998 30219 tablet_bootstrap.cc:492] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704: No bootstrap required, opened a new log
I20260926 22:43:51.272069 30219 ts_tablet_manager.cc:1397] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:51.272543 30222 tablet_bootstrap.cc:492] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a: Bootstrap starting.
I20260926 22:43:51.272746 30218 raft_consensus.cc:348] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b48c6fcf7c4dbd84af0c4139534b2a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 37185 } } peers { permanent_uuid: "e130bca81c1a46c18e85464127093a6b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 34377 } } peers { permanent_uuid: "5f530f831e914ab4a0adb4ac264e5704" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33927 } }
I20260926 22:43:51.272928 30218 raft_consensus.cc:374] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:51.272966 30218 raft_consensus.cc:729] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e130bca81c1a46c18e85464127093a6b, State: Initialized, Role: FOLLOWER
I20260926 22:43:51.273103 30218 consensus_queue.cc:260] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b48c6fcf7c4dbd84af0c4139534b2a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 37185 } } peers { permanent_uuid: "e130bca81c1a46c18e85464127093a6b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 34377 } } peers { permanent_uuid: "5f530f831e914ab4a0adb4ac264e5704" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33927 } }
I20260926 22:43:51.273180 30222 tablet_bootstrap.cc:654] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:51.273384 30218 ts_tablet_manager.cc:1428] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:43:51.273505 30204 heartbeater.cc:503] Master 127.11.107.254:33775 was elected leader, sending a full tablet report...
I20260926 22:43:51.273756 30222 log.cc:824] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:51.274346 30219 raft_consensus.cc:348] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b48c6fcf7c4dbd84af0c4139534b2a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 37185 } } peers { permanent_uuid: "e130bca81c1a46c18e85464127093a6b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 34377 } } peers { permanent_uuid: "5f530f831e914ab4a0adb4ac264e5704" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33927 } }
I20260926 22:43:51.274490 30222 tablet_bootstrap.cc:492] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a: No bootstrap required, opened a new log
I20260926 22:43:51.274526 30219 raft_consensus.cc:374] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:51.274539 30222 ts_tablet_manager.cc:1397] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:51.274573 30219 raft_consensus.cc:729] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f530f831e914ab4a0adb4ac264e5704, State: Initialized, Role: FOLLOWER
I20260926 22:43:51.274700 30219 consensus_queue.cc:260] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b48c6fcf7c4dbd84af0c4139534b2a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 37185 } } peers { permanent_uuid: "e130bca81c1a46c18e85464127093a6b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 34377 } } peers { permanent_uuid: "5f530f831e914ab4a0adb4ac264e5704" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33927 } }
I20260926 22:43:51.274959 30219 ts_tablet_manager.cc:1428] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:43:51.274969 30070 heartbeater.cc:503] Master 127.11.107.254:33775 was elected leader, sending a full tablet report...
I20260926 22:43:51.276592 30222 raft_consensus.cc:348] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b48c6fcf7c4dbd84af0c4139534b2a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 37185 } } peers { permanent_uuid: "e130bca81c1a46c18e85464127093a6b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 34377 } } peers { permanent_uuid: "5f530f831e914ab4a0adb4ac264e5704" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33927 } }
I20260926 22:43:51.276717 30222 raft_consensus.cc:374] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:51.276739 30222 raft_consensus.cc:729] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49b48c6fcf7c4dbd84af0c4139534b2a, State: Initialized, Role: FOLLOWER
I20260926 22:43:51.276852 30222 consensus_queue.cc:260] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b48c6fcf7c4dbd84af0c4139534b2a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 37185 } } peers { permanent_uuid: "e130bca81c1a46c18e85464127093a6b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 34377 } } peers { permanent_uuid: "5f530f831e914ab4a0adb4ac264e5704" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33927 } }
I20260926 22:43:51.277112 30222 ts_tablet_manager.cc:1428] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:43:51.277145 29936 heartbeater.cc:503] Master 127.11.107.254:33775 was elected leader, sending a full tablet report...
W20260926 22:43:51.326735 30071 tablet.cc:2369] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:43:51.432271 29937 tablet.cc:2369] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:43:51.476233 30205 tablet.cc:2369] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:43:51.490561 30225 raft_consensus.cc:482] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:51.490679 30225 raft_consensus.cc:504] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b48c6fcf7c4dbd84af0c4139534b2a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 37185 } } peers { permanent_uuid: "e130bca81c1a46c18e85464127093a6b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 34377 } } peers { permanent_uuid: "5f530f831e914ab4a0adb4ac264e5704" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33927 } }
I20260926 22:43:51.491071 30225 leader_election.cc:288] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49b48c6fcf7c4dbd84af0c4139534b2a (127.11.107.193:37185), e130bca81c1a46c18e85464127093a6b (127.11.107.195:34377)
I20260926 22:43:51.494027 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.491281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:37185 (local address 127.11.107.194:54351)
0926 22:43:51.491340 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:51.491352 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:51.491423 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:51.491609 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:51.491610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:51.491611 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:51.491694 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.491697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.492437 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.492438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.493205 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.493207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.493744 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.493745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.493821 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.493824 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:43:51.493915 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 22:43:51.493956 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 22:43:51.494036 30214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.491381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:54351 (local address 127.11.107.193:37185)
0926 22:43:51.491434 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:51.491436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:51.491450 (+ 14us) server_negotiation.cc:407] Connection header received
0926 22:43:51.491484 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:51.491485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:51.491494 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:51.491554 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:51.491800 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.492353 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.493301 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.493675 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.493729 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.493735 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:51.493932 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 22:43:51.493965 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 22:43:51.494228 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.491423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:34377 (local address 127.11.107.194:44835)
0926 22:43:51.491625 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:51.491641 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:51.491686 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:51.491813 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:51.491813 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:51.491815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:51.491888 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.491889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.492505 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.492505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.493238 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.493263 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.493953 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.493954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.494028 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.494032 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:43:51.494090 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 22:43:51.494142 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":114,"threads_started":1}
I20260926 22:43:51.494366 30217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.491502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:44835 (local address 127.11.107.195:34377)
0926 22:43:51.491561 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:51.491563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:51.491699 (+ 136us) server_negotiation.cc:407] Connection header received
0926 22:43:51.491728 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:51.491729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:51.491740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:51.491795 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:51.491954 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.492427 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.493387 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.493884 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.493953 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.493961 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:51.494263 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 22:43:51.494306 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 22:43:51.494427 29890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdff6d8698e4f3f9c5cae19b6c90f1c" candidate_uuid: "5f530f831e914ab4a0adb4ac264e5704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49b48c6fcf7c4dbd84af0c4139534b2a" is_pre_election: true
I20260926 22:43:51.494586 29890 raft_consensus.cc:2454] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f530f831e914ab4a0adb4ac264e5704 in term 0.
I20260926 22:43:51.494829 29959 leader_election.cc:302] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49b48c6fcf7c4dbd84af0c4139534b2a, 5f530f831e914ab4a0adb4ac264e5704; no voters:
I20260926 22:43:51.494944 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdff6d8698e4f3f9c5cae19b6c90f1c" candidate_uuid: "5f530f831e914ab4a0adb4ac264e5704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e130bca81c1a46c18e85464127093a6b" is_pre_election: true
I20260926 22:43:51.494978 30225 raft_consensus.cc:2781] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:43:51.495045 30225 raft_consensus.cc:482] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:43:51.495085 30225 raft_consensus.cc:3037] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:51.495087 30158 raft_consensus.cc:2454] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f530f831e914ab4a0adb4ac264e5704 in term 0.
I20260926 22:43:51.495879 30225 raft_consensus.cc:504] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b48c6fcf7c4dbd84af0c4139534b2a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 37185 } } peers { permanent_uuid: "e130bca81c1a46c18e85464127093a6b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 34377 } } peers { permanent_uuid: "5f530f831e914ab4a0adb4ac264e5704" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33927 } }
I20260926 22:43:51.496016 30225 leader_election.cc:288] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [CANDIDATE]: Term 1 election: Requested vote from peers 49b48c6fcf7c4dbd84af0c4139534b2a (127.11.107.193:37185), e130bca81c1a46c18e85464127093a6b (127.11.107.195:34377)
I20260926 22:43:51.496220 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdff6d8698e4f3f9c5cae19b6c90f1c" candidate_uuid: "5f530f831e914ab4a0adb4ac264e5704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e130bca81c1a46c18e85464127093a6b"
I20260926 22:43:51.496229 29890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdff6d8698e4f3f9c5cae19b6c90f1c" candidate_uuid: "5f530f831e914ab4a0adb4ac264e5704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49b48c6fcf7c4dbd84af0c4139534b2a"
I20260926 22:43:51.496294 30158 raft_consensus.cc:3037] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:51.496318 29890 raft_consensus.cc:3037] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:51.497026 30158 raft_consensus.cc:2454] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f530f831e914ab4a0adb4ac264e5704 in term 1.
I20260926 22:43:51.497030 29890 raft_consensus.cc:2454] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f530f831e914ab4a0adb4ac264e5704 in term 1.
I20260926 22:43:51.497211 29959 leader_election.cc:302] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49b48c6fcf7c4dbd84af0c4139534b2a, 5f530f831e914ab4a0adb4ac264e5704; no voters:
I20260926 22:43:51.497328 30225 raft_consensus.cc:2781] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:51.497490 30225 raft_consensus.cc:686] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [term 1 LEADER]: Becoming Leader. State: Replica: 5f530f831e914ab4a0adb4ac264e5704, State: Running, Role: LEADER
I20260926 22:43:51.497589 30225 consensus_queue.cc:237] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b48c6fcf7c4dbd84af0c4139534b2a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 37185 } } peers { permanent_uuid: "e130bca81c1a46c18e85464127093a6b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 34377 } } peers { permanent_uuid: "5f530f831e914ab4a0adb4ac264e5704" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33927 } }
I20260926 22:43:51.498417 29744 catalog_manager.cc:5495] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f530f831e914ab4a0adb4ac264e5704 (127.11.107.194). New cstate: current_term: 1 leader_uuid: "5f530f831e914ab4a0adb4ac264e5704" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b48c6fcf7c4dbd84af0c4139534b2a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e130bca81c1a46c18e85464127093a6b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f530f831e914ab4a0adb4ac264e5704" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 33927 } health_report { overall_health: HEALTHY } } }
I20260926 22:43:51.545663 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.543111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.11.107.194:33927)
0926 22:43:51.543175 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:51.543177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:51.543185 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:43:51.543223 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:51.543224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:51.543242 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:51.543317 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:43:51.543455 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.543927 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.544844 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.545050 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.545120 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.545289 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:43:51.545374 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:43:51.545448 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:51.545576 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 22:43:51.545586 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 22:43:51.547623 29890 raft_consensus.cc:1261] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 49b48c6fcf7c4dbd84af0c4139534b2a [term 1 FOLLOWER]: Refusing update from remote peer 5f530f831e914ab4a0adb4ac264e5704: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:51.547616 30158 raft_consensus.cc:1261] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b [term 1 FOLLOWER]: Refusing update from remote peer 5f530f831e914ab4a0adb4ac264e5704: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:51.547974 30228 consensus_queue.cc:1035] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49b48c6fcf7c4dbd84af0c4139534b2a" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:51.548142 30225 consensus_queue.cc:1035] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e130bca81c1a46c18e85464127093a6b" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:51.550629 30232 mvcc.cc:205] Tried to move back new op lower bound from 7333734938815918080 to 7333734938614640640. Current Snapshot: MvccSnapshot[applied={T|T < 7333734938815918080}]
I20260926 22:43:51.551484 30233 mvcc.cc:205] Tried to move back new op lower bound from 7333734938815918080 to 7333734938614640640. Current Snapshot: MvccSnapshot[applied={T|T < 7333734938815918080 or (T in {7333734938815918080})}]
I20260926 22:43:51.552585 29745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:51.552690 29745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:51.553392 29745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77383cece6f7429993fb77b2d62f5c5c: Sending DeleteTablet for 3 replicas of tablet 8fdff6d8698e4f3f9c5cae19b6c90f1c
I20260926 22:43:51.553736 30004 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdff6d8698e4f3f9c5cae19b6c90f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:51 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:55812
I20260926 22:43:51.553814 29870 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdff6d8698e4f3f9c5cae19b6c90f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:51 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:51126
I20260926 22:43:51.554055 30241 tablet_replica.cc:330] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704: stopping tablet replica
I20260926 22:43:51.554050 30138 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdff6d8698e4f3f9c5cae19b6c90f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:51 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:35748
I20260926 22:43:51.554112 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 29807
I20260926 22:43:51.554152 30241 raft_consensus.cc:2229] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:43:51.554708 30241 raft_consensus.cc:2258] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:51.555167 30241 ts_tablet_manager.cc:1905] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:51.557192 30241 ts_tablet_manager.cc:1918] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:51.557256 30241 log.cc:1196] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/8fdff6d8698e4f3f9c5cae19b6c90f1c
I20260926 22:43:51.557492 30241 ts_tablet_manager.cc:1939] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P 5f530f831e914ab4a0adb4ac264e5704: Deleting consensus metadata
I20260926 22:43:51.557945 29732 catalog_manager.cc:4841] TS 5f530f831e914ab4a0adb4ac264e5704 (127.11.107.194:33927): tablet 8fdff6d8698e4f3f9c5cae19b6c90f1c (table test-table [id=06455b255e2c483a898b3202e8eb7eb0]) successfully deleted
W20260926 22:43:51.561816 29731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:37185 (error 108)
I20260926 22:43:51.561995 30243 tablet_replica.cc:330] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b: stopping tablet replica
I20260926 22:43:51.562119 30243 raft_consensus.cc:2229] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:43:51.562252 30243 raft_consensus.cc:2258] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:51.562407 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.562283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:37185 (local address 127.0.0.1:51132)
0926 22:43:51.562340 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:51.562373 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:37185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 22:43:51.562469 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 29941
W20260926 22:43:51.562515 29731 catalog_manager.cc:4584] TS 49b48c6fcf7c4dbd84af0c4139534b2a (127.11.107.193:37185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fdff6d8698e4f3f9c5cae19b6c90f1c: Network error: Client connection negotiation failed: client connection to 127.11.107.193:37185: connect: Connection refused (error 111)
I20260926 22:43:51.563474 30243 ts_tablet_manager.cc:1905] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:51.564859 30243 ts_tablet_manager.cc:1918] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:51.564946 30243 log.cc:1196] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/8fdff6d8698e4f3f9c5cae19b6c90f1c
I20260926 22:43:51.565204 30243 ts_tablet_manager.cc:1939] T 8fdff6d8698e4f3f9c5cae19b6c90f1c P e130bca81c1a46c18e85464127093a6b: Deleting consensus metadata
I20260926 22:43:51.565599 29731 catalog_manager.cc:4841] TS e130bca81c1a46c18e85464127093a6b (127.11.107.195:34377): tablet 8fdff6d8698e4f3f9c5cae19b6c90f1c (table test-table [id=06455b255e2c483a898b3202e8eb7eb0]) successfully deleted
I20260926 22:43:51.567196 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 30075
I20260926 22:43:51.571517 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 29713
2026-09-26T22:43:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2107 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T22:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:43:51Z Disabled control of system clock
I20260926 22:43:51.598479 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:37609
--webserver_interface=127.11.107.254
--webserver_port=0
--builtin_ntp_servers=127.11.107.212:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:37609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:43:51.681509 30248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:51.681856 30248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:51.681890 30248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:51.683046 30248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:43:51.683117 30248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:51.683157 30248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:51.683185 30248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:43:51.683210 30248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:43:51.684502 30248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:37609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:37609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:51.684777 30248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:51.685058 30248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:51.689785 30254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:51.689788 30253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:51.689916 30256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:51.689947 30248 server_base.cc:1034] running on GCE node
I20260926 22:43:51.690404 30248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:51.690714 30248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:51.691892 30248 hybrid_clock.cc:648] HybridClock initialized: now 1790462631691878 us; error 27 us; skew 500 ppm
I20260926 22:43:51.693167 30248 webserver.cc:460] Webserver started at http://127.11.107.254:43697/ using document root <none> and password file <none>
I20260926 22:43:51.693413 30248 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:51.693461 30248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:51.693609 30248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:51.694752 30248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "99058da7ef1e4cdeb6a98ffe5a02e147"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "6b90bdbeda48a55b356a9486d62f639d"
server_key_iv: "c7afe607817cc3928de9f5a311919160"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.695111 30248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "99058da7ef1e4cdeb6a98ffe5a02e147"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "6b90bdbeda48a55b356a9486d62f639d"
server_key_iv: "c7afe607817cc3928de9f5a311919160"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.696357 30248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:43:51.697110 30262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:51.697326 30248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:51.697424 30248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "99058da7ef1e4cdeb6a98ffe5a02e147"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "6b90bdbeda48a55b356a9486d62f639d"
server_key_iv: "c7afe607817cc3928de9f5a311919160"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.697512 30248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:51.719867 30248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:51.720249 30248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:51.720415 30248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:51.724797 30248 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:37609
I20260926 22:43:51.724870 30314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:37609 every 8 connection(s)
I20260926 22:43:51.725237 30248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:43:51.726295 30315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:51.728544 30315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147: Bootstrap starting.
I20260926 22:43:51.729089 30315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:51.729326 30315 log.cc:824] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:51.730010 30315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147: No bootstrap required, opened a new log
I20260926 22:43:51.731516 30315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99058da7ef1e4cdeb6a98ffe5a02e147" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37609 } }
I20260926 22:43:51.731685 30315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:51.731710 30315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99058da7ef1e4cdeb6a98ffe5a02e147, State: Initialized, Role: FOLLOWER
I20260926 22:43:51.731818 30315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99058da7ef1e4cdeb6a98ffe5a02e147" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37609 } }
I20260926 22:43:51.731871 30315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:43:51.731892 30315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:43:51.731966 30315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:51.732575 30315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99058da7ef1e4cdeb6a98ffe5a02e147" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37609 } }
I20260926 22:43:51.732604 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 30248
I20260926 22:43:51.732725 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:43:51.732685 30315 leader_election.cc:302] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99058da7ef1e4cdeb6a98ffe5a02e147; no voters:
I20260926 22:43:51.732867 30315 leader_election.cc:288] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:43:51.732909 30318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:51.732933 11695 external_mini_cluster.cc:1428] Setting key 41ba9794f0628f711f40beacfc0549b7
I20260926 22:43:51.733036 30318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [term 1 LEADER]: Becoming Leader. State: Replica: 99058da7ef1e4cdeb6a98ffe5a02e147, State: Running, Role: LEADER
I20260926 22:43:51.733145 30318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99058da7ef1e4cdeb6a98ffe5a02e147" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37609 } }
I20260926 22:43:51.733551 30315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:43:51.733592 30320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99058da7ef1e4cdeb6a98ffe5a02e147. Latest consensus state: current_term: 1 leader_uuid: "99058da7ef1e4cdeb6a98ffe5a02e147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99058da7ef1e4cdeb6a98ffe5a02e147" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37609 } } }
I20260926 22:43:51.733592 30319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99058da7ef1e4cdeb6a98ffe5a02e147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99058da7ef1e4cdeb6a98ffe5a02e147" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 37609 } } }
I20260926 22:43:51.733724 30320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:51.734012 30324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:43:51.733768 30319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:51.734632 30324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:43:51.736204 30324 catalog_manager.cc:1347] Generated new cluster ID: b17df42ef62d4dbea0f7883ece5cf307
I20260926 22:43:51.736248 30324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:43:51.739854 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.733423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.11.107.254:37609)
0926 22:43:51.733627 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:51.733636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:51.734116 (+ 480us) server_negotiation.cc:407] Connection header received
0926 22:43:51.734601 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:51.734608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:51.734687 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:51.734809 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:51.735180 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.736550 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.737624 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.738282 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.738414 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.739136 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:51.739140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:51.739141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:51.739354 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:51.739396 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:51.739410 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:51.739482 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:51.739548 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 22:43:51.739640 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260926 22:43:51.762487 30324 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:43:51.763059 30324 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:43:51.773348 30324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147: Generated new TSK 0
I20260926 22:43:51.773556 30324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:43:51.776818 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:37609
--builtin_ntp_servers=127.11.107.212:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:43:51.856983 30339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:51.857180 30339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:51.857203 30339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:51.858340 30339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:51.858376 30339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:51.858400 30339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:43:51.859709 30339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:37609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:51.859879 30339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:51.860107 30339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:51.865047 30344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:51.865033 30347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:51.865032 30345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:51.865140 30339 server_base.cc:1034] running on GCE node
I20260926 22:43:51.865438 30339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:51.865686 30339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:51.866846 30339 hybrid_clock.cc:648] HybridClock initialized: now 1790462631866825 us; error 33 us; skew 500 ppm
I20260926 22:43:51.871169 30339 webserver.cc:460] Webserver started at http://127.11.107.193:45613/ using document root <none> and password file <none>
I20260926 22:43:51.871408 30339 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:51.871475 30339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:51.871651 30339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:51.872776 30339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "08229a6a76db4c79aa6b34eab5ec0575"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "eb6a6cbf63e7acef752545d675cd60cc"
server_key_iv: "fd4250a1ba161974991140d24f95331f"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.873126 30339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "08229a6a76db4c79aa6b34eab5ec0575"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "eb6a6cbf63e7acef752545d675cd60cc"
server_key_iv: "fd4250a1ba161974991140d24f95331f"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.874503 30339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:43:51.875236 30353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:51.875412 30339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:43:51.875510 30339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "08229a6a76db4c79aa6b34eab5ec0575"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "eb6a6cbf63e7acef752545d675cd60cc"
server_key_iv: "fd4250a1ba161974991140d24f95331f"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.875617 30339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:51.887494 30339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:51.887761 30339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:51.887902 30339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:51.888110 30339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:51.888419 30339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:51.888451 30339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:51.888490 30339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:51.888515 30339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:51.894093 30339 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:41123
I20260926 22:43:51.894165 30466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:41123 every 8 connection(s)
I20260926 22:43:51.894472 30339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:43:51.899458 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.895170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:52271 (local address 127.11.107.254:37609)
0926 22:43:51.895220 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:51.895222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:51.895443 (+ 221us) server_negotiation.cc:407] Connection header received
0926 22:43:51.895868 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:51.895870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:51.895885 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:51.895955 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:51.896562 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.897095 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.898154 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.898372 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.898436 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.899201 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:51.899203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:51.899203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:51.899208 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:51.899222 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:51.899226 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:51.899277 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:51.899365 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 22:43:51.899394 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:43:51.899652 30469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:51.895119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:37609 (local address 127.11.107.193:52271)
0926 22:43:51.895294 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:51.895332 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:51.895783 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:51.896002 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:51.896005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:51.896013 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:51.896489 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.896494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.897195 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.897196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.898062 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:51.898065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:51.898449 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:51.898449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:51.898551 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:51.899044 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:43:51.899045 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:51.899087 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:43:51.899110 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:51.899290 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:51.899294 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:51.899296 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:43:51.899354 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 22:43:51.899433 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260926 22:43:51.900243 30467 heartbeater.cc:347] Connected to a master server at 127.11.107.254:37609
I20260926 22:43:51.900336 30467 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:51.900512 30467 heartbeater.cc:511] Master 127.11.107.254:37609 requested a full tablet report, sending...
I20260926 22:43:51.901031 30279 ts_manager.cc:194] Registered new tserver with Master: 08229a6a76db4c79aa6b34eab5ec0575 (127.11.107.193:41123)
I20260926 22:43:51.901432 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 30339
I20260926 22:43:51.901539 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:43:51.901736 11695 external_mini_cluster.cc:1428] Setting key c140469549cd86c55f0f6ffc5fe74ae6
I20260926 22:43:51.901962 30279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.107.193:52271
I20260926 22:43:51.902804 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:37609
--builtin_ntp_servers=127.11.107.212:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:43:51.984006 30470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:51.984193 30470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:51.984213 30470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:51.985337 30470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:51.985375 30470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:51.985399 30470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:43:51.986821 30470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:37609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:51.987052 30470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:51.987340 30470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:51.992163 30476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:51.992209 30475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:51.992185 30478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:51.992235 30470 server_base.cc:1034] running on GCE node
I20260926 22:43:51.992664 30470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:51.992928 30470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:51.994081 30470 hybrid_clock.cc:648] HybridClock initialized: now 1790462631994080 us; error 43 us; skew 500 ppm
I20260926 22:43:51.995395 30470 webserver.cc:460] Webserver started at http://127.11.107.194:38649/ using document root <none> and password file <none>
I20260926 22:43:51.995656 30470 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:51.995703 30470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:51.995862 30470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:51.996934 30470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "ff59b2428025412399cb43f31c0352ab"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "2323549b29adefcc80e3843ecf09815e"
server_key_iv: "740012db4eeff37e3979a23cdf72c557"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.997277 30470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "ff59b2428025412399cb43f31c0352ab"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "2323549b29adefcc80e3843ecf09815e"
server_key_iv: "740012db4eeff37e3979a23cdf72c557"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.998662 30470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:43:51.999419 30484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:51.999651 30470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:43:51.999758 30470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "ff59b2428025412399cb43f31c0352ab"
format_stamp: "Formatted at 2026-09-26 22:43:51 on dist-test-slave-j8sq"
server_key: "2323549b29adefcc80e3843ecf09815e"
server_key_iv: "740012db4eeff37e3979a23cdf72c557"
server_key_version: "encryptionkey@0"
I20260926 22:43:51.999830 30470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:52.030277 30470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:52.030606 30470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:52.030756 30470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:52.030997 30470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:52.031332 30470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:52.031389 30470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:52.031432 30470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:52.031461 30470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:52.037416 30470 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:34777
I20260926 22:43:52.037487 30597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:34777 every 8 connection(s)
I20260926 22:43:52.037871 30470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:43:52.043300 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.038671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:33419 (local address 127.11.107.254:37609)
0926 22:43:52.038732 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:52.038734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:52.038973 (+ 239us) server_negotiation.cc:407] Connection header received
0926 22:43:52.039524 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:52.039526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:52.039538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:52.039586 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:52.040323 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.040825 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.041796 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.042065 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.042140 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.042980 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:52.042982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:52.042982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:52.042986 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:52.043004 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:52.043013 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:52.043076 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:52.043192 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 22:43:52.043222 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:43:52.043537 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.038651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:37609 (local address 127.11.107.194:33419)
0926 22:43:52.038850 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:52.038882 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:52.039405 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:52.039680 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:52.039687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:52.039701 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:52.040197 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.040201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.040930 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.040931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.041709 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.041711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.042149 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.042150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.042277 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.042806 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:43:52.042806 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:52.042835 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:43:52.042867 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:52.043078 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:52.043084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:52.043087 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:43:52.043189 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 22:43:52.043289 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260926 22:43:52.044034 30598 heartbeater.cc:347] Connected to a master server at 127.11.107.254:37609
I20260926 22:43:52.044108 30598 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:52.044317 30598 heartbeater.cc:511] Master 127.11.107.254:37609 requested a full tablet report, sending...
I20260926 22:43:52.044720 30278 ts_manager.cc:194] Registered new tserver with Master: ff59b2428025412399cb43f31c0352ab (127.11.107.194:34777)
I20260926 22:43:52.045096 30278 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.107.194:33419
I20260926 22:43:52.047554 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 30470
I20260926 22:43:52.047631 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:43:52.047798 11695 external_mini_cluster.cc:1428] Setting key 09097eb10387c5e6aac9ae14e523ab74
I20260926 22:43:52.048609 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:37609
--builtin_ntp_servers=127.11.107.212:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:43:52.129510 30601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:52.129729 30601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:52.129751 30601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:52.130896 30601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:52.130941 30601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:52.130968 30601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:43:52.132215 30601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:37609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:52.132423 30601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:52.132681 30601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:43:52.137805 30601 server_base.cc:1034] running on GCE node
W20260926 22:43:52.137743 30609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:52.137789 30606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:52.137735 30607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:52.138136 30601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:52.138417 30601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:52.139564 30601 hybrid_clock.cc:648] HybridClock initialized: now 1790462632139538 us; error 39 us; skew 500 ppm
I20260926 22:43:52.140842 30601 webserver.cc:460] Webserver started at http://127.11.107.195:33571/ using document root <none> and password file <none>
I20260926 22:43:52.141108 30601 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:52.141157 30601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:52.141331 30601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:52.142458 30601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "fecea49bbc5c47d9a435c06b81a30b14"
format_stamp: "Formatted at 2026-09-26 22:43:52 on dist-test-slave-j8sq"
server_key: "5657f328cded6a926a9335a5fcaf3f1b"
server_key_iv: "d9d4b7a2639f0d7fba5c674af5d6e4bc"
server_key_version: "encryptionkey@0"
I20260926 22:43:52.142831 30601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "fecea49bbc5c47d9a435c06b81a30b14"
format_stamp: "Formatted at 2026-09-26 22:43:52 on dist-test-slave-j8sq"
server_key: "5657f328cded6a926a9335a5fcaf3f1b"
server_key_iv: "d9d4b7a2639f0d7fba5c674af5d6e4bc"
server_key_version: "encryptionkey@0"
I20260926 22:43:52.144155 30601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:43:52.144857 30615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:52.145015 30601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:43:52.145116 30601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "fecea49bbc5c47d9a435c06b81a30b14"
format_stamp: "Formatted at 2026-09-26 22:43:52 on dist-test-slave-j8sq"
server_key: "5657f328cded6a926a9335a5fcaf3f1b"
server_key_iv: "d9d4b7a2639f0d7fba5c674af5d6e4bc"
server_key_version: "encryptionkey@0"
I20260926 22:43:52.145190 30601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:52.152529 30601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:52.152778 30601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:52.152936 30601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:52.153134 30601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:52.153427 30601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:52.153460 30601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:52.153479 30601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:52.153512 30601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:52.159004 30601 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:36367
I20260926 22:43:52.159077 30728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:36367 every 8 connection(s)
I20260926 22:43:52.159369 30601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:43:52.163174 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 30601
I20260926 22:43:52.163308 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:43:52.163520 11695 external_mini_cluster.cc:1428] Setting key 7c7dd902e7c740b840b91f8fd6851531
I20260926 22:43:52.164733 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.160058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:55513 (local address 127.11.107.254:37609)
0926 22:43:52.160102 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:52.160104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:52.160347 (+ 243us) server_negotiation.cc:407] Connection header received
0926 22:43:52.160806 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:52.160807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:52.160818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:52.160886 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:52.161528 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.162148 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.163096 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.163333 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.163422 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.164431 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:52.164432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:52.164433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:52.164436 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:52.164450 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:52.164454 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:52.164500 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:52.164599 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 22:43:52.164631 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:43:52.164949 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.160013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:37609 (local address 127.11.107.195:55513)
0926 22:43:52.160201 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:52.160239 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:52.160674 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:52.160955 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:52.160959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:52.160967 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:52.161447 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.161450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.162223 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.162224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.162988 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.162990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.163428 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.163430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.163603 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.164265 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:43:52.164266 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:52.164302 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:43:52.164324 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:52.164515 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:52.164519 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:52.164522 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:43:52.164590 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 22:43:52.164704 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260926 22:43:52.165443 30729 heartbeater.cc:347] Connected to a master server at 127.11.107.254:37609
I20260926 22:43:52.165519 30729 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:52.165652 30729 heartbeater.cc:511] Master 127.11.107.254:37609 requested a full tablet report, sending...
I20260926 22:43:52.165990 30278 ts_manager.cc:194] Registered new tserver with Master: fecea49bbc5c47d9a435c06b81a30b14 (127.11.107.195:36367)
I20260926 22:43:52.166334 30278 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.107.195:55513
I20260926 22:43:52.175649 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:43:52.180270 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.177004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.11.107.254:37609)
0926 22:43:52.177084 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:52.177086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:52.177193 (+ 107us) server_negotiation.cc:407] Connection header received
0926 22:43:52.177271 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:52.177272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:52.177284 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:52.177356 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:52.177516 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.178079 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.178914 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.179168 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.179242 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.179972 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:52.179974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:52.179974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:52.179978 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:52.179992 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:52.179996 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:52.180111 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:52.180177 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 22:43:52.180187 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 22:43:52.182078 30278 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:43:52.183547 30278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:43:52.189996 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.185935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.11.107.194:34777)
0926 22:43:52.186271 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:52.186275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:52.186283 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:43:52.186355 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:52.186356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:52.186373 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:52.186450 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:52.186689 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.187788 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.189019 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.189562 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.189667 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.189759 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:52.189868 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 22:43:52.189909 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":88,"threads_started":1}
I20260926 22:43:52.189998 30738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.185816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:41123 (local address 127.0.0.1:44966)
0926 22:43:52.186150 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:52.186162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:52.186222 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:52.186553 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:52.186554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:52.186556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:52.186613 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.186615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.187572 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.187573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.188742 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.188744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.189702 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.189703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.189765 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.189770 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:43:52.189801 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 22:43:52.189903 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":101,"threads_started":1}
I20260926 22:43:52.189994 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.185795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:34777 (local address 127.0.0.1:48560)
0926 22:43:52.186012 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:52.186049 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:52.186115 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:52.186530 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:52.186533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:52.186536 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:52.186598 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.186601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.187883 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.187883 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.188926 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.188929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.189664 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.189665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.189743 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.189755 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 22:43:52.189854 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 22:43:52.189898 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":30,"thread_start_us":87,"threads_started":1}
I20260926 22:43:52.190223 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.185817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:36367 (local address 127.0.0.1:49776)
0926 22:43:52.186353 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:52.186365 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:52.186442 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:52.186720 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:52.186720 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:52.186722 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:52.186755 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.186756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.187987 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.187988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.189027 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.189029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.190024 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.190025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.190077 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.190080 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:43:52.190128 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 22:43:52.190166 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":145,"threads_started":1}
I20260926 22:43:52.190305 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.186060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.11.107.193:41123)
0926 22:43:52.186365 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:52.186370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:52.186383 (+ 13us) server_negotiation.cc:407] Connection header received
0926 22:43:52.186428 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:52.186431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:52.186448 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:52.186692 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:52.186753 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.187467 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.188838 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.189591 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.190039 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.190141 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:52.190165 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 22:43:52.190211 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":80,"threads_started":1}
I20260926 22:43:52.190645 30742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.186105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.11.107.195:36367)
0926 22:43:52.186298 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:52.186303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:52.186417 (+ 114us) server_negotiation.cc:407] Connection header received
0926 22:43:52.186542 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:52.186544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:52.186573 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:52.186629 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:52.186839 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.187892 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.189119 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.189813 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.189903 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.189990 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:52.190316 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 22:43:52.190558 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260926 22:43:52.190979 30401 tablet_service.cc:1467] Processing CreateTablet for tablet dc0fc81e0f4b44aea398c6420f6324d1 (DEFAULT_TABLE table=test-table [id=28ef14d31632416e8810919c55796da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:52.191366 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0fc81e0f4b44aea398c6420f6324d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:52.191332 30532 tablet_service.cc:1467] Processing CreateTablet for tablet dc0fc81e0f4b44aea398c6420f6324d1 (DEFAULT_TABLE table=test-table [id=28ef14d31632416e8810919c55796da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:52.191387 30663 tablet_service.cc:1467] Processing CreateTablet for tablet dc0fc81e0f4b44aea398c6420f6324d1 (DEFAULT_TABLE table=test-table [id=28ef14d31632416e8810919c55796da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:52.191650 30532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0fc81e0f4b44aea398c6420f6324d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:52.191781 30663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0fc81e0f4b44aea398c6420f6324d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:52.194541 30743 tablet_bootstrap.cc:492] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14: Bootstrap starting.
I20260926 22:43:52.195019 30744 tablet_bootstrap.cc:492] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575: Bootstrap starting.
I20260926 22:43:52.195302 30743 tablet_bootstrap.cc:654] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:52.195658 30743 log.cc:824] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:52.195749 30744 tablet_bootstrap.cc:654] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:52.196061 30744 log.cc:824] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:52.196416 30745 tablet_bootstrap.cc:492] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab: Bootstrap starting.
I20260926 22:43:52.196444 30743 tablet_bootstrap.cc:492] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14: No bootstrap required, opened a new log
I20260926 22:43:52.196512 30743 ts_tablet_manager.cc:1397] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:43:52.196796 30744 tablet_bootstrap.cc:492] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575: No bootstrap required, opened a new log
I20260926 22:43:52.196877 30744 ts_tablet_manager.cc:1397] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:43:52.197101 30745 tablet_bootstrap.cc:654] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:52.197433 30745 log.cc:824] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:52.198285 30745 tablet_bootstrap.cc:492] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab: No bootstrap required, opened a new log
I20260926 22:43:52.198361 30745 ts_tablet_manager.cc:1397] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:43:52.199039 30744 raft_consensus.cc:348] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff59b2428025412399cb43f31c0352ab" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34777 } } peers { permanent_uuid: "fecea49bbc5c47d9a435c06b81a30b14" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 36367 } } peers { permanent_uuid: "08229a6a76db4c79aa6b34eab5ec0575" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41123 } }
I20260926 22:43:52.199196 30744 raft_consensus.cc:374] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:52.199239 30744 raft_consensus.cc:729] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08229a6a76db4c79aa6b34eab5ec0575, State: Initialized, Role: FOLLOWER
I20260926 22:43:52.199350 30744 consensus_queue.cc:260] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff59b2428025412399cb43f31c0352ab" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34777 } } peers { permanent_uuid: "fecea49bbc5c47d9a435c06b81a30b14" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 36367 } } peers { permanent_uuid: "08229a6a76db4c79aa6b34eab5ec0575" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41123 } }
I20260926 22:43:52.199612 30744 ts_tablet_manager.cc:1428] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 22:43:52.199760 30467 heartbeater.cc:503] Master 127.11.107.254:37609 was elected leader, sending a full tablet report...
I20260926 22:43:52.200083 30743 raft_consensus.cc:348] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff59b2428025412399cb43f31c0352ab" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34777 } } peers { permanent_uuid: "fecea49bbc5c47d9a435c06b81a30b14" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 36367 } } peers { permanent_uuid: "08229a6a76db4c79aa6b34eab5ec0575" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41123 } }
I20260926 22:43:52.200255 30743 raft_consensus.cc:374] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:52.200349 30743 raft_consensus.cc:729] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fecea49bbc5c47d9a435c06b81a30b14, State: Initialized, Role: FOLLOWER
I20260926 22:43:52.200488 30743 consensus_queue.cc:260] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff59b2428025412399cb43f31c0352ab" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34777 } } peers { permanent_uuid: "fecea49bbc5c47d9a435c06b81a30b14" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 36367 } } peers { permanent_uuid: "08229a6a76db4c79aa6b34eab5ec0575" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41123 } }
I20260926 22:43:52.200762 30743 ts_tablet_manager.cc:1428] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 22:43:52.200804 30729 heartbeater.cc:503] Master 127.11.107.254:37609 was elected leader, sending a full tablet report...
I20260926 22:43:52.201050 30745 raft_consensus.cc:348] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff59b2428025412399cb43f31c0352ab" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34777 } } peers { permanent_uuid: "fecea49bbc5c47d9a435c06b81a30b14" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 36367 } } peers { permanent_uuid: "08229a6a76db4c79aa6b34eab5ec0575" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41123 } }
I20260926 22:43:52.201229 30745 raft_consensus.cc:374] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:52.201283 30745 raft_consensus.cc:729] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff59b2428025412399cb43f31c0352ab, State: Initialized, Role: FOLLOWER
I20260926 22:43:52.201395 30745 consensus_queue.cc:260] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff59b2428025412399cb43f31c0352ab" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34777 } } peers { permanent_uuid: "fecea49bbc5c47d9a435c06b81a30b14" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 36367 } } peers { permanent_uuid: "08229a6a76db4c79aa6b34eab5ec0575" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41123 } }
I20260926 22:43:52.201648 30745 ts_tablet_manager.cc:1428] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 22:43:52.201764 30598 heartbeater.cc:503] Master 127.11.107.254:37609 was elected leader, sending a full tablet report...
W20260926 22:43:52.288452 30599 tablet.cc:2369] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:43:52.395151 30468 tablet.cc:2369] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:43:52.409967 30730 tablet.cc:2369] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:43:52.531344 30749 raft_consensus.cc:482] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:52.531471 30749 raft_consensus.cc:504] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff59b2428025412399cb43f31c0352ab" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34777 } } peers { permanent_uuid: "fecea49bbc5c47d9a435c06b81a30b14" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 36367 } } peers { permanent_uuid: "08229a6a76db4c79aa6b34eab5ec0575" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41123 } }
I20260926 22:43:52.531814 30749 leader_election.cc:288] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff59b2428025412399cb43f31c0352ab (127.11.107.194:34777), fecea49bbc5c47d9a435c06b81a30b14 (127.11.107.195:36367)
I20260926 22:43:52.535221 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.531987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:34777 (local address 127.11.107.193:51735)
0926 22:43:52.532138 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:52.532149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:52.532236 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:52.532422 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:52.532424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:52.532426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:52.532505 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.532508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.533302 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.533303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.534294 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.534296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.534927 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.534928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.534997 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.535001 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:43:52.535068 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 22:43:52.535106 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260926 22:43:52.535221 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.532055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:51735 (local address 127.11.107.194:34777)
0926 22:43:52.532132 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:52.532135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:52.532207 (+ 72us) server_negotiation.cc:407] Connection header received
0926 22:43:52.532336 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:52.532338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:52.532351 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:52.532424 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:52.532586 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.533198 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.534379 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.534851 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.534911 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.534919 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:52.535074 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 22:43:52.535100 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 22:43:52.535517 30742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.532142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:37335 (local address 127.11.107.195:36367)
0926 22:43:52.532198 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:52.532200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:52.532246 (+ 46us) server_negotiation.cc:407] Connection header received
0926 22:43:52.532345 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:52.532346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:52.532361 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:52.532433 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:52.532660 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.533178 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.534670 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.535120 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.535200 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.535207 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:52.535417 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 22:43:52.535458 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:43:52.535561 30552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0fc81e0f4b44aea398c6420f6324d1" candidate_uuid: "08229a6a76db4c79aa6b34eab5ec0575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff59b2428025412399cb43f31c0352ab" is_pre_election: true
I20260926 22:43:52.535749 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.532031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:36367 (local address 127.11.107.193:37335)
0926 22:43:52.532176 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:52.532187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:52.532258 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:52.532447 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:52.532448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:52.532450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:52.532531 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.532532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.533257 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.533259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.534550 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:52.534589 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.535241 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.535241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:52.535331 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.535336 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:43:52.535608 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 22:43:52.535656 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260926 22:43:52.535761 30552 raft_consensus.cc:2454] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08229a6a76db4c79aa6b34eab5ec0575 in term 0.
I20260926 22:43:52.536015 30355 leader_election.cc:302] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08229a6a76db4c79aa6b34eab5ec0575, ff59b2428025412399cb43f31c0352ab; no voters:
I20260926 22:43:52.536155 30749 raft_consensus.cc:2781] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:43:52.536247 30749 raft_consensus.cc:482] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:43:52.536271 30749 raft_consensus.cc:3037] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:52.536346 30683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0fc81e0f4b44aea398c6420f6324d1" candidate_uuid: "08229a6a76db4c79aa6b34eab5ec0575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fecea49bbc5c47d9a435c06b81a30b14" is_pre_election: true
I20260926 22:43:52.536489 30683 raft_consensus.cc:2454] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08229a6a76db4c79aa6b34eab5ec0575 in term 0.
I20260926 22:43:52.537207 30749 raft_consensus.cc:504] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff59b2428025412399cb43f31c0352ab" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34777 } } peers { permanent_uuid: "fecea49bbc5c47d9a435c06b81a30b14" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 36367 } } peers { permanent_uuid: "08229a6a76db4c79aa6b34eab5ec0575" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41123 } }
I20260926 22:43:52.537343 30749 leader_election.cc:288] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [CANDIDATE]: Term 1 election: Requested vote from peers ff59b2428025412399cb43f31c0352ab (127.11.107.194:34777), fecea49bbc5c47d9a435c06b81a30b14 (127.11.107.195:36367)
I20260926 22:43:52.537499 30552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0fc81e0f4b44aea398c6420f6324d1" candidate_uuid: "08229a6a76db4c79aa6b34eab5ec0575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff59b2428025412399cb43f31c0352ab"
I20260926 22:43:52.537500 30683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0fc81e0f4b44aea398c6420f6324d1" candidate_uuid: "08229a6a76db4c79aa6b34eab5ec0575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fecea49bbc5c47d9a435c06b81a30b14"
I20260926 22:43:52.537590 30552 raft_consensus.cc:3037] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:52.537590 30683 raft_consensus.cc:3037] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:52.538533 30683 raft_consensus.cc:2454] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08229a6a76db4c79aa6b34eab5ec0575 in term 1.
I20260926 22:43:52.538533 30552 raft_consensus.cc:2454] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08229a6a76db4c79aa6b34eab5ec0575 in term 1.
I20260926 22:43:52.538698 30355 leader_election.cc:302] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08229a6a76db4c79aa6b34eab5ec0575, ff59b2428025412399cb43f31c0352ab; no voters:
I20260926 22:43:52.538789 30749 raft_consensus.cc:2781] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:52.538924 30749 raft_consensus.cc:686] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [term 1 LEADER]: Becoming Leader. State: Replica: 08229a6a76db4c79aa6b34eab5ec0575, State: Running, Role: LEADER
I20260926 22:43:52.539031 30749 consensus_queue.cc:237] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff59b2428025412399cb43f31c0352ab" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34777 } } peers { permanent_uuid: "fecea49bbc5c47d9a435c06b81a30b14" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 36367 } } peers { permanent_uuid: "08229a6a76db4c79aa6b34eab5ec0575" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41123 } }
I20260926 22:43:52.539953 30279 catalog_manager.cc:5495] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08229a6a76db4c79aa6b34eab5ec0575 (127.11.107.193). New cstate: current_term: 1 leader_uuid: "08229a6a76db4c79aa6b34eab5ec0575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff59b2428025412399cb43f31c0352ab" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fecea49bbc5c47d9a435c06b81a30b14" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08229a6a76db4c79aa6b34eab5ec0575" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 41123 } health_report { overall_health: HEALTHY } } }
I20260926 22:43:52.628820 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.626555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.11.107.193:41123)
0926 22:43:52.626606 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:52.626608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:52.626616 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:43:52.626699 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:52.626700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:52.626720 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:52.626778 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:43:52.626930 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.627401 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.628131 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:52.628319 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:52.628375 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:52.628512 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:43:52.628626 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:43:52.628700 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:52.628752 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 22:43:52.628761 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:43:52.630820 30683 raft_consensus.cc:1261] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14 [term 1 FOLLOWER]: Refusing update from remote peer 08229a6a76db4c79aa6b34eab5ec0575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:52.630820 30552 raft_consensus.cc:1261] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab [term 1 FOLLOWER]: Refusing update from remote peer 08229a6a76db4c79aa6b34eab5ec0575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:52.631201 30754 consensus_queue.cc:1035] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fecea49bbc5c47d9a435c06b81a30b14" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:52.631343 30749 consensus_queue.cc:1035] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff59b2428025412399cb43f31c0352ab" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:52.633661 30757 mvcc.cc:205] Tried to move back new op lower bound from 7333734943252512768 to 7333734942880645120. Current Snapshot: MvccSnapshot[applied={T|T < 7333734943252512768}]
I20260926 22:43:52.636456 30279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:52.636546 30279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:52.637367 30279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99058da7ef1e4cdeb6a98ffe5a02e147: Sending DeleteTablet for 3 replicas of tablet dc0fc81e0f4b44aea398c6420f6324d1
I20260926 22:43:52.637724 30401 tablet_service.cc:1514] Processing DeleteTablet for tablet dc0fc81e0f4b44aea398c6420f6324d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:52 UTC) from {username='slave'} at 127.0.0.1:44966
I20260926 22:43:52.637773 30532 tablet_service.cc:1514] Processing DeleteTablet for tablet dc0fc81e0f4b44aea398c6420f6324d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:52 UTC) from {username='slave'} at 127.0.0.1:48560
I20260926 22:43:52.637971 30663 tablet_service.cc:1514] Processing DeleteTablet for tablet dc0fc81e0f4b44aea398c6420f6324d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:52 UTC) from {username='slave'} at 127.0.0.1:49776
I20260926 22:43:52.638172 30769 tablet_replica.cc:330] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14: stopping tablet replica
I20260926 22:43:52.638237 30767 tablet_replica.cc:330] T dc0fc81e0f4b44aea398c6420f6324d1 P 08229a6a76db4c79aa6b34eab5ec0575: stopping tablet replica
I20260926 22:43:52.638240 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 30339
I20260926 22:43:52.638893 30768 tablet_replica.cc:330] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab: stopping tablet replica
I20260926 22:43:52.639034 30768 raft_consensus.cc:2229] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:43:52.639231 30768 raft_consensus.cc:2258] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:52.639724 30768 ts_tablet_manager.cc:1905] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:52.638329 30769 raft_consensus.cc:2229] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:43:52.640385 30769 raft_consensus.cc:2258] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:52.640820 30769 ts_tablet_manager.cc:1905] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:52.641702 30768 ts_tablet_manager.cc:1918] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:52.641781 30768 log.cc:1196] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/dc0fc81e0f4b44aea398c6420f6324d1
I20260926 22:43:52.642091 30769 ts_tablet_manager.cc:1918] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:52.642103 30768 ts_tablet_manager.cc:1939] T dc0fc81e0f4b44aea398c6420f6324d1 P ff59b2428025412399cb43f31c0352ab: Deleting consensus metadata
I20260926 22:43:52.642179 30769 log.cc:1196] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/dc0fc81e0f4b44aea398c6420f6324d1
I20260926 22:43:52.642407 30769 ts_tablet_manager.cc:1939] T dc0fc81e0f4b44aea398c6420f6324d1 P fecea49bbc5c47d9a435c06b81a30b14: Deleting consensus metadata
I20260926 22:43:52.642499 30264 catalog_manager.cc:4841] TS ff59b2428025412399cb43f31c0352ab (127.11.107.194:34777): tablet dc0fc81e0f4b44aea398c6420f6324d1 (table test-table [id=28ef14d31632416e8810919c55796da1]) successfully deleted
I20260926 22:43:52.642841 30265 catalog_manager.cc:4841] TS fecea49bbc5c47d9a435c06b81a30b14 (127.11.107.195:36367): tablet dc0fc81e0f4b44aea398c6420f6324d1 (table test-table [id=28ef14d31632416e8810919c55796da1]) successfully deleted
W20260926 22:43:52.643703 30266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:41123 (error 108)
I20260926 22:43:52.644165 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:52.644037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:41123 (local address 127.0.0.1:44992)
0926 22:43:52.644089 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:52.644126 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:41123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 22:43:52.644153 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 30470
W20260926 22:43:52.644277 30266 catalog_manager.cc:4584] TS 08229a6a76db4c79aa6b34eab5ec0575 (127.11.107.193:41123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc0fc81e0f4b44aea398c6420f6324d1: Network error: Client connection negotiation failed: client connection to 127.11.107.193:41123: connect: Connection refused (error 111)
I20260926 22:43:52.649117 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 30601
I20260926 22:43:52.653456 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 30248
2026-09-26T22:43:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1081 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:43:52 dist-test-slave-j8sq krb5kdc[30773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:43:52 dist-test-slave-j8sq krb5kdc[30773](info): set up 2 sockets
Sep 26 22:43:52 dist-test-slave-j8sq krb5kdc[30773](info): commencing operation
krb5kdc: starting...
W20260926 22:43:53.683630 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:43:53 dist-test-slave-j8sq krb5kdc[30773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:43:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:43:53.798794 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:45463
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:34423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:45463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:53.877113 30786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:53.877342 30786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:53.877378 30786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:53.878510 30786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:43:53.878574 30786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:53.878599 30786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:53.878625 30786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:43:53.878643 30786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:43:53.880049 30786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:34423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:45463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:45463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:53.880321 30786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:53.880544 30786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:53.885222 30794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:53.885198 30791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:53.885237 30792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:53.885366 30786 server_base.cc:1034] running on GCE node
I20260926 22:43:53.885677 30786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:53.885996 30786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:53.887233 30786 hybrid_clock.cc:648] HybridClock initialized: now 1790462633887213 us; error 29 us; skew 500 ppm
Sep 26 22:43:53 dist-test-slave-j8sq krb5kdc[30773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462633, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:53.889621 30786 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:43:53.890089 30786 webserver.cc:460] Webserver started at http://127.11.107.254:35723/ using document root <none> and password file <none>
I20260926 22:43:53.890313 30786 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:53.890377 30786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:53.890503 30786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:53.891492 30786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "8bfddeea37644ebfab48f51010983eb7"
format_stamp: "Formatted at 2026-09-26 22:43:53 on dist-test-slave-j8sq"
server_key: "f6c15da2177772d35908004aa0f23f5e"
server_key_iv: "0ea19dc0b05748b12650d4de60bfd49f"
server_key_version: "encryptionkey@0"
I20260926 22:43:53.891855 30786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "8bfddeea37644ebfab48f51010983eb7"
format_stamp: "Formatted at 2026-09-26 22:43:53 on dist-test-slave-j8sq"
server_key: "f6c15da2177772d35908004aa0f23f5e"
server_key_iv: "0ea19dc0b05748b12650d4de60bfd49f"
server_key_version: "encryptionkey@0"
I20260926 22:43:53.893204 30786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:43:53.893945 30801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:53.894109 30786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:53.894202 30786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "8bfddeea37644ebfab48f51010983eb7"
format_stamp: "Formatted at 2026-09-26 22:43:53 on dist-test-slave-j8sq"
server_key: "f6c15da2177772d35908004aa0f23f5e"
server_key_iv: "0ea19dc0b05748b12650d4de60bfd49f"
server_key_version: "encryptionkey@0"
I20260926 22:43:53.894280 30786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:53.909106 30786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:53.910876 30786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:53.911024 30786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:53.914763 30786 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:45463
I20260926 22:43:53.914798 30853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:45463 every 8 connection(s)
I20260926 22:43:53.915139 30786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:43:53.915999 30854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:53.918241 30854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7: Bootstrap starting.
I20260926 22:43:53.918792 30854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:53.919075 30854 log.cc:824] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:53.919763 30854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7: No bootstrap required, opened a new log
I20260926 22:43:53.921279 30854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bfddeea37644ebfab48f51010983eb7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45463 } }
I20260926 22:43:53.921406 30854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:53.921427 30854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bfddeea37644ebfab48f51010983eb7, State: Initialized, Role: FOLLOWER
I20260926 22:43:53.921595 30854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bfddeea37644ebfab48f51010983eb7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45463 } }
I20260926 22:43:53.921657 30854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:43:53.921679 30854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:43:53.921730 30854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:53.922283 30854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bfddeea37644ebfab48f51010983eb7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45463 } }
I20260926 22:43:53.922371 30854 leader_election.cc:302] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bfddeea37644ebfab48f51010983eb7; no voters:
I20260926 22:43:53.922540 30854 leader_election.cc:288] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:43:53.922631 30857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:53.922778 30854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:43:53.922801 30857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [term 1 LEADER]: Becoming Leader. State: Replica: 8bfddeea37644ebfab48f51010983eb7, State: Running, Role: LEADER
I20260926 22:43:53.922991 30857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bfddeea37644ebfab48f51010983eb7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45463 } }
I20260926 22:43:53.923094 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 30786
I20260926 22:43:53.923192 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:43:53.923393 11695 external_mini_cluster.cc:1428] Setting key dceb77883d5d58f973222a608ad81574
I20260926 22:43:53.923388 30859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bfddeea37644ebfab48f51010983eb7. Latest consensus state: current_term: 1 leader_uuid: "8bfddeea37644ebfab48f51010983eb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bfddeea37644ebfab48f51010983eb7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45463 } } }
I20260926 22:43:53.923475 30859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:53.923503 30858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bfddeea37644ebfab48f51010983eb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bfddeea37644ebfab48f51010983eb7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45463 } } }
I20260926 22:43:53.923574 30858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:53.924309 30866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:43:53.924863 30866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:43:53.926306 30866 catalog_manager.cc:1347] Generated new cluster ID: 6a54efc393fa4002a3f3067808766ade
I20260926 22:43:53.926362 30866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:43:53 dist-test-slave-j8sq krb5kdc[30773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:53.947028 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:53.924114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.11.107.254:45463)
0926 22:43:53.924322 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:53.924331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:53.924364 (+ 33us) server_negotiation.cc:407] Connection header received
0926 22:43:53.924540 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:53.924547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:53.924603 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:53.924760 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:53.925609 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:53.938324 (+ 12715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:53.939437 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:53.940045 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:53.940142 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:53.942888 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:53.942892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:53.942892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:53.942904 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:53.944837 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:53.945231 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:53.945234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:53.945235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:53.945342 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:53.945541 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:53.945542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:53.945543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:53.945858 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:53.946010 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:53.946121 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 22:43:53.946259 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260926 22:43:53.957124 30866 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:43:53.957612 30866 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:43:53.961359 30866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7: Generated new TSK 0
I20260926 22:43:53.961520 30866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.107.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
Entry for principal kudu/127.11.107.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab.
I20260926 22:43:53.992915 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:45463
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:34423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:54.073522 30880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:54.073721 30880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:54.073745 30880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:54.074909 30880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:54.074952 30880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:54.074977 30880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:43:54.076412 30880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:34423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.193.keytab
--principal=kudu/127.11.107.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:45463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:54.076649 30880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:54.076884 30880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:54.081746 30886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:54.081753 30888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:54.081760 30885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:54.081959 30880 server_base.cc:1034] running on GCE node
I20260926 22:43:54.082161 30880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:54.082425 30880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:54.083599 30880 hybrid_clock.cc:648] HybridClock initialized: now 1790462634083567 us; error 41 us; skew 500 ppm
Sep 26 22:43:54 dist-test-slave-j8sq krb5kdc[30773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:54.086090 30880 init.cc:377] Logged in from keytab as kudu/127.11.107.193@KRBTEST.COM (short username kudu)
I20260926 22:43:54.086503 30880 webserver.cc:460] Webserver started at http://127.11.107.193:37827/ using document root <none> and password file <none>
I20260926 22:43:54.086738 30880 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:54.086807 30880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:54.086961 30880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:54.087996 30880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "43be633e889e4cf0b844fb1bce8b9384"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "924916846000d5a1e19716092b3e0691"
server_key_iv: "d154939417e98c4ebbcc6ec5adccd47e"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.088361 30880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "43be633e889e4cf0b844fb1bce8b9384"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "924916846000d5a1e19716092b3e0691"
server_key_iv: "d154939417e98c4ebbcc6ec5adccd47e"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.089710 30880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:43:54.090561 30895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:54.090790 30880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:54.090900 30880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "43be633e889e4cf0b844fb1bce8b9384"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "924916846000d5a1e19716092b3e0691"
server_key_iv: "d154939417e98c4ebbcc6ec5adccd47e"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.090991 30880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:54.133870 30880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:54.136065 30880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:54.136238 30880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:54.136488 30880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:54.136808 30880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:54.136868 30880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:54.136909 30880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:54.136940 30880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:54.142632 30880 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:46621
I20260926 22:43:54.142753 31008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:46621 every 8 connection(s)
I20260926 22:43:54.143036 30880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:43:54.147676 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 30880
I20260926 22:43:54.147802 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:43:54.147998 11695 external_mini_cluster.cc:1428] Setting key b8633cae4a2aff8bcbbd3c2301142cbb
Sep 26 22:43:54 dist-test-slave-j8sq krb5kdc[30773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.193@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:54.153184 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.143746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:37657 (local address 127.11.107.254:45463)
0926 22:43:54.143818 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:54.143820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:54.144088 (+ 268us) server_negotiation.cc:407] Connection header received
0926 22:43:54.144618 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:54.144619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:54.144631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:54.144689 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:54.145630 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.146202 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.147227 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.147423 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.147476 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.150099 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:54.150101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:54.150102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:54.150109 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:54.151685 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:54.152127 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:54.152128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:54.152129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:54.152195 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:54.152482 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:54.152483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:54.152483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:54.152604 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:54.152692 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:54.152978 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 22:43:54.153088 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 22:43:54.153383 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.143728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45463 (local address 127.11.107.193:37657)
0926 22:43:54.143922 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:54.143980 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:54.144525 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:54.144749 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:54.144753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:54.145353 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:54.145547 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.145552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.146280 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.146280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.147144 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.147146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.147513 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.147514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.147614 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.148194 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:54.148202 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:54.149984 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:54.151796 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:54.151804 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:54.151806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:54.152046 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:54.152271 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:54.152272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:54.152273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:54.152401 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:54.152684 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:54.152691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:54.152836 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 22:43:54.152955 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 22:43:54.153115 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
I20260926 22:43:54.153954 31009 heartbeater.cc:347] Connected to a master server at 127.11.107.254:45463
I20260926 22:43:54.154062 31009 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:54.154247 31009 heartbeater.cc:511] Master 127.11.107.254:45463 requested a full tablet report, sending...
I20260926 22:43:54.154835 30818 ts_manager.cc:194] Registered new tserver with Master: 43be633e889e4cf0b844fb1bce8b9384 (127.11.107.193:46621)
I20260926 22:43:54.155479 30818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.193@KRBTEST.COM'} at 127.11.107.193:37657
WARNING: no policy specified for kudu/127.11.107.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
Entry for principal kudu/127.11.107.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab.
I20260926 22:43:54.172955 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:45463
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:34423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:54.252791 31014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:54.252993 31014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:54.253017 31014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:54.254225 31014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:54.254268 31014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:54.254297 31014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:43:54.255776 31014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:34423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.194.keytab
--principal=kudu/127.11.107.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:45463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:54.256017 31014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:54.256253 31014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:54.261413 31022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:54.261413 31019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:54.261485 31020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:54.261502 31014 server_base.cc:1034] running on GCE node
I20260926 22:43:54.261942 31014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:54.262187 31014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:54.263334 31014 hybrid_clock.cc:648] HybridClock initialized: now 1790462634263322 us; error 27 us; skew 500 ppm
Sep 26 22:43:54 dist-test-slave-j8sq krb5kdc[30773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:54.265751 31014 init.cc:377] Logged in from keytab as kudu/127.11.107.194@KRBTEST.COM (short username kudu)
I20260926 22:43:54.266266 31014 webserver.cc:460] Webserver started at http://127.11.107.194:41395/ using document root <none> and password file <none>
I20260926 22:43:54.266502 31014 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:54.266571 31014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:54.266721 31014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:54.267792 31014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "2d1548184ad5498b8063ff77a25feb8f"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "1c6d3bb808947f77e5a9db8ac714d9fd"
server_key_iv: "646b10dba7c7a85ab1fa6b08ab168fc4"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.268141 31014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "2d1548184ad5498b8063ff77a25feb8f"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "1c6d3bb808947f77e5a9db8ac714d9fd"
server_key_iv: "646b10dba7c7a85ab1fa6b08ab168fc4"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.269407 31014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:43:54.270182 31029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:54.270350 31014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:43:54.270453 31014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "2d1548184ad5498b8063ff77a25feb8f"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "1c6d3bb808947f77e5a9db8ac714d9fd"
server_key_iv: "646b10dba7c7a85ab1fa6b08ab168fc4"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.270524 31014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:54.287259 31014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:54.289109 31014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:54.289270 31014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:54.289490 31014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:54.289825 31014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:54.289933 31014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:54.289973 31014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:54.290000 31014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:54.295892 31014 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:43275
I20260926 22:43:54.295938 31142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:43275 every 8 connection(s)
I20260926 22:43:54.296298 31014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:43:54.297809 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 31014
I20260926 22:43:54.297966 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:43:54.298187 11695 external_mini_cluster.cc:1428] Setting key 3647119222be555dcf83f1a0ed3ef3d7
Sep 26 22:43:54 dist-test-slave-j8sq krb5kdc[30773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.194@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:54.306967 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.296968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:36999 (local address 127.11.107.254:45463)
0926 22:43:54.297009 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:54.297011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:54.297277 (+ 266us) server_negotiation.cc:407] Connection header received
0926 22:43:54.297881 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:54.297883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:54.297896 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:54.297981 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:54.299168 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.299950 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.301043 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.301332 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.301401 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.303854 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:54.303856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:54.303857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:54.303864 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:54.305499 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:54.305963 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:54.305965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:54.305965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:54.306025 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:54.306273 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:54.306274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:54.306274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:54.306425 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:54.306503 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:54.306765 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 22:43:54.306854 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:43:54.306990 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.296951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45463 (local address 127.11.107.194:36999)
0926 22:43:54.297143 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:54.297177 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:54.297732 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:54.298093 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:54.298097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:54.298856 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:54.299082 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.299087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.300067 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.300068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.300921 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.300924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.301416 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.301417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.301528 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.302059 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:54.302068 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:54.303757 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:54.305587 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:54.305593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:54.305595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:54.305806 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:54.306106 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:54.306107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:54.306107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:54.306203 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:54.306501 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:54.306506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:54.306625 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 22:43:54.306712 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 22:43:54.306825 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.11.107.195@KRBTEST.COM; defaulting to no policy
I20260926 22:43:54.307591 31143 heartbeater.cc:347] Connected to a master server at 127.11.107.254:45463
I20260926 22:43:54.307682 31143 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:54.307816 31143 heartbeater.cc:511] Master 127.11.107.254:45463 requested a full tablet report, sending...
I20260926 22:43:54.308199 30818 ts_manager.cc:194] Registered new tserver with Master: 2d1548184ad5498b8063ff77a25feb8f (127.11.107.194:43275)
I20260926 22:43:54.308590 30818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.194@KRBTEST.COM'} at 127.11.107.194:36999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
Entry for principal kudu/127.11.107.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab.
I20260926 22:43:54.325152 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:45463
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:34423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:54.404469 31148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:54.404697 31148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:54.404734 31148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:54.405939 31148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:54.405999 31148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:54.406044 31148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:43:54.407469 31148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:34423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.195.keytab
--principal=kudu/127.11.107.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:45463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:54.407742 31148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:54.407970 31148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:43:54.412957 31148 server_base.cc:1034] running on GCE node
W20260926 22:43:54.412930 31156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:54.412914 31154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:54.412930 31153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:54.413316 31148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:54.413558 31148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:54.414700 31148 hybrid_clock.cc:648] HybridClock initialized: now 1790462634414688 us; error 29 us; skew 500 ppm
Sep 26 22:43:54 dist-test-slave-j8sq krb5kdc[30773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:54.417086 31148 init.cc:377] Logged in from keytab as kudu/127.11.107.195@KRBTEST.COM (short username kudu)
I20260926 22:43:54.417549 31148 webserver.cc:460] Webserver started at http://127.11.107.195:37837/ using document root <none> and password file <none>
I20260926 22:43:54.417783 31148 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:54.417876 31148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:54.418056 31148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:54.419067 31148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "dd9a4de4b1a00d4122738699816b7f24"
server_key_iv: "0ddf7c047fd90715e8d358d8f5e07603"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.419415 31148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "dd9a4de4b1a00d4122738699816b7f24"
server_key_iv: "0ddf7c047fd90715e8d358d8f5e07603"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.420709 31148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:43:54.421414 31163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:54.421653 31148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:43:54.421741 31148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "dd9a4de4b1a00d4122738699816b7f24"
server_key_iv: "0ddf7c047fd90715e8d358d8f5e07603"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.421808 31148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:54.457050 31148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:54.459215 31148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:54.459357 31148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:54.459594 31148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:54.459900 31148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:54.459933 31148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:54.459973 31148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:54.460000 31148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:54.465960 31148 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:38735
I20260926 22:43:54.466065 31276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:38735 every 8 connection(s)
I20260926 22:43:54.466364 31148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:43:54.470165 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 31148
I20260926 22:43:54.470273 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:43:54.470460 11695 external_mini_cluster.cc:1428] Setting key f7b067ce9b8a276b0859acb3ab41550e
Sep 26 22:43:54 dist-test-slave-j8sq krb5kdc[30773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.195@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:43:54.476413 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.467129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:60617 (local address 127.11.107.254:45463)
0926 22:43:54.467209 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:54.467211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:54.467399 (+ 188us) server_negotiation.cc:407] Connection header received
0926 22:43:54.467935 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:54.467936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:54.467946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:54.467997 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:54.468966 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.469452 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.470995 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.471316 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.471384 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.473804 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:54.473805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:54.473806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:54.473812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:54.475269 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:54.475681 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:54.475682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:54.475682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:54.475731 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:54.475912 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:54.475913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:54.475913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:54.476006 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:54.476054 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:54.476294 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 22:43:54.476350 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 22:43:54.476706 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.467100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45463 (local address 127.11.107.195:60617)
0926 22:43:54.467275 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:54.467312 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:54.467846 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:54.468054 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:54.468057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:54.468685 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:54.468886 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.468890 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.469536 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.469537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.470915 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.470918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.471416 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.471418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.471558 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.472132 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:43:54.472141 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:54.473723 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:54.475346 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:54.475352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:54.475354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:54.475613 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:54.475774 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:43:54.475774 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:43:54.475775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:43:54.475863 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:43:54.476066 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:54.476070 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:54.476183 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 22:43:54.476271 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 22:43:54.476389 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260926 22:43:54.477236 31277 heartbeater.cc:347] Connected to a master server at 127.11.107.254:45463
I20260926 22:43:54.477344 31277 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:54.477519 31277 heartbeater.cc:511] Master 127.11.107.254:45463 requested a full tablet report, sending...
I20260926 22:43:54.477932 30818 ts_manager.cc:194] Registered new tserver with Master: 0369c4c5d6bc46a9b628d3ea1b7bcdbd (127.11.107.195:38735)
I20260926 22:43:54.478276 30818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.107.195@KRBTEST.COM'} at 127.11.107.195:60617
I20260926 22:43:54.482236 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:43:54.489022 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.483519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.11.107.254:45463)
0926 22:43:54.483589 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:54.483591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:54.483667 (+ 76us) server_negotiation.cc:407] Connection header received
0926 22:43:54.483700 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:54.483701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:54.483709 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:54.483752 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:54.484246 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.484686 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.485383 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.485560 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.485611 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.486839 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:54.486840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:54.486840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:54.486845 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:54.488012 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:54.488345 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:54.488346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:54.488347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:54.488394 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:54.488555 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:54.488556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:54.488556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:54.488693 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:54.488767 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:54.488865 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 22:43:54.488920 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 22:43:54.490564 30818 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:43:54.492676 30818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:43:54.499081 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.494879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:46621 (local address 127.0.0.1:39630)
0926 22:43:54.495343 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:54.495351 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:54.495398 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:54.495643 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:54.495644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:54.495646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:54.495696 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.495698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.496963 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.496964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.498118 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.498120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.498793 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.498794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.498871 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.498883 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 22:43:54.498950 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 22:43:54.498998 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":32,"thread_start_us":89,"threads_started":1}
I20260926 22:43:54.499182 31290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.494872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:38735 (local address 127.0.0.1:37810)
0926 22:43:54.495135 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:54.495176 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:54.495242 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:54.495553 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:54.495555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:54.495557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:54.495614 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.495616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.496992 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.496993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.498046 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.498048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.498931 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.498932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.498986 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.498989 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:43:54.499039 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 22:43:54.499069 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":54,"threads_started":1}
I20260926 22:43:54.499186 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.495009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.11.107.195:38735)
0926 22:43:54.495368 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:54.495371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:54.495381 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:43:54.495419 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:54.495421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:54.495438 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:54.495705 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:54.495765 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.496869 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.498151 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.498852 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.498932 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.498982 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:54.499049 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 22:43:54.499085 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":77,"threads_started":1}
I20260926 22:43:54.499509 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.495051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.11.107.194:43275)
0926 22:43:54.495252 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:54.495257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:54.495408 (+ 151us) server_negotiation.cc:407] Connection header received
0926 22:43:54.495449 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:54.495451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:54.495466 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:54.495533 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:54.495756 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.496542 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.498095 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.498852 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.498937 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.498982 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:54.499370 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 22:43:54.499418 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":93,"threads_started":1}
I20260926 22:43:54.499665 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.495138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39630 (local address 127.11.107.193:46621)
0926 22:43:54.495321 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:54.495325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:54.495392 (+ 67us) server_negotiation.cc:407] Connection header received
0926 22:43:54.495462 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:54.495465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:54.495479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:54.495556 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:54.495781 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.496877 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.498201 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.498698 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.498784 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.498844 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:54.499466 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 22:43:54.499526 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260926 22:43:54.499894 31211 tablet_service.cc:1467] Processing CreateTablet for tablet 32f925eca7564758a01553255116350a (DEFAULT_TABLE table=test-table [id=1dedeb8c56ac4bbb8680d249e1a8a4d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:54.500025 31288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.494876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:43275 (local address 127.0.0.1:33684)
0926 22:43:54.495293 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:54.495303 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:54.495352 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:54.495604 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:54.495605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:54.495606 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:54.495679 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.495681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.496659 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.496660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.498012 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.498015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.499014 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.499015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.499082 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.499086 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:43:54.499622 (+ 536us) client_negotiation.cc:240] Negotiation successful
0926 22:43:54.499685 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":23,"thread_start_us":93,"threads_started":1}
I20260926 22:43:54.500361 31211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f925eca7564758a01553255116350a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:54.500379 30943 tablet_service.cc:1467] Processing CreateTablet for tablet 32f925eca7564758a01553255116350a (DEFAULT_TABLE table=test-table [id=1dedeb8c56ac4bbb8680d249e1a8a4d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:54.500713 30943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f925eca7564758a01553255116350a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:54.503870 31295 tablet_bootstrap.cc:492] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384: Bootstrap starting.
I20260926 22:43:54.504052 31294 tablet_bootstrap.cc:492] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd: Bootstrap starting.
I20260926 22:43:54.504462 31077 tablet_service.cc:1467] Processing CreateTablet for tablet 32f925eca7564758a01553255116350a (DEFAULT_TABLE table=test-table [id=1dedeb8c56ac4bbb8680d249e1a8a4d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:54.504602 31295 tablet_bootstrap.cc:654] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:54.504650 31294 tablet_bootstrap.cc:654] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:54.504771 31077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f925eca7564758a01553255116350a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:54.505234 31294 log.cc:824] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:54.505234 31295 log.cc:824] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:54.506042 31295 tablet_bootstrap.cc:492] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384: No bootstrap required, opened a new log
I20260926 22:43:54.506186 31295 ts_tablet_manager.cc:1397] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:54.506247 31294 tablet_bootstrap.cc:492] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd: No bootstrap required, opened a new log
I20260926 22:43:54.506431 31294 ts_tablet_manager.cc:1397] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 22:43:54.507615 31298 tablet_bootstrap.cc:492] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f: Bootstrap starting.
I20260926 22:43:54.508193 31298 tablet_bootstrap.cc:654] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:54.508139 31295 raft_consensus.cc:348] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1548184ad5498b8063ff77a25feb8f" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43275 } } peers { permanent_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38735 } } peers { permanent_uuid: "43be633e889e4cf0b844fb1bce8b9384" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 46621 } }
I20260926 22:43:54.508314 31295 raft_consensus.cc:374] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:54.508360 31295 raft_consensus.cc:729] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43be633e889e4cf0b844fb1bce8b9384, State: Initialized, Role: FOLLOWER
I20260926 22:43:54.508345 31294 raft_consensus.cc:348] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1548184ad5498b8063ff77a25feb8f" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43275 } } peers { permanent_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38735 } } peers { permanent_uuid: "43be633e889e4cf0b844fb1bce8b9384" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 46621 } }
I20260926 22:43:54.508497 31294 raft_consensus.cc:374] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:54.508473 31295 consensus_queue.cc:260] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1548184ad5498b8063ff77a25feb8f" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43275 } } peers { permanent_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38735 } } peers { permanent_uuid: "43be633e889e4cf0b844fb1bce8b9384" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 46621 } }
I20260926 22:43:54.508534 31294 raft_consensus.cc:729] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0369c4c5d6bc46a9b628d3ea1b7bcdbd, State: Initialized, Role: FOLLOWER
I20260926 22:43:54.508677 31294 consensus_queue.cc:260] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1548184ad5498b8063ff77a25feb8f" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43275 } } peers { permanent_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38735 } } peers { permanent_uuid: "43be633e889e4cf0b844fb1bce8b9384" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 46621 } }
I20260926 22:43:54.508731 31295 ts_tablet_manager.cc:1428] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:54.508745 31298 log.cc:824] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:54.508848 31009 heartbeater.cc:503] Master 127.11.107.254:45463 was elected leader, sending a full tablet report...
I20260926 22:43:54.508948 31294 ts_tablet_manager.cc:1428] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:54.509109 31277 heartbeater.cc:503] Master 127.11.107.254:45463 was elected leader, sending a full tablet report...
I20260926 22:43:54.509490 31298 tablet_bootstrap.cc:492] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f: No bootstrap required, opened a new log
I20260926 22:43:54.509553 31298 ts_tablet_manager.cc:1397] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:43:54.511160 31298 raft_consensus.cc:348] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1548184ad5498b8063ff77a25feb8f" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43275 } } peers { permanent_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38735 } } peers { permanent_uuid: "43be633e889e4cf0b844fb1bce8b9384" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 46621 } }
I20260926 22:43:54.511283 31298 raft_consensus.cc:374] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:54.511304 31298 raft_consensus.cc:729] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d1548184ad5498b8063ff77a25feb8f, State: Initialized, Role: FOLLOWER
I20260926 22:43:54.511400 31298 consensus_queue.cc:260] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1548184ad5498b8063ff77a25feb8f" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43275 } } peers { permanent_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38735 } } peers { permanent_uuid: "43be633e889e4cf0b844fb1bce8b9384" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 46621 } }
I20260926 22:43:54.511615 31298 ts_tablet_manager.cc:1428] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:54.511667 31143 heartbeater.cc:503] Master 127.11.107.254:45463 was elected leader, sending a full tablet report...
W20260926 22:43:54.546842 31144 tablet.cc:2369] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:43:54.594362 31300 raft_consensus.cc:482] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:54.594529 31300 raft_consensus.cc:504] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1548184ad5498b8063ff77a25feb8f" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43275 } } peers { permanent_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38735 } } peers { permanent_uuid: "43be633e889e4cf0b844fb1bce8b9384" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 46621 } }
I20260926 22:43:54.594877 31300 leader_election.cc:288] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d1548184ad5498b8063ff77a25feb8f (127.11.107.194:43275), 43be633e889e4cf0b844fb1bce8b9384 (127.11.107.193:46621)
I20260926 22:43:54.597843 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.595062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:46621 (local address 127.11.107.195:38367)
0926 22:43:54.595237 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:54.595246 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:54.595301 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:54.595447 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:54.595448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:54.595450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:54.595517 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.595518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.596153 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.596154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.596993 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.596995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.597570 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.597571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.597654 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.597659 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:43:54.597709 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 22:43:54.597739 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":110,"threads_started":1}
I20260926 22:43:54.597886 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.595150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:38367 (local address 127.11.107.193:46621)
0926 22:43:54.595193 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:54.595195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:54.595312 (+ 117us) server_negotiation.cc:407] Connection header received
0926 22:43:54.595345 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:54.595346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:54.595358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:54.595418 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:54.595585 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.596074 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.597126 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.597495 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.597548 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.597555 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:54.597733 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 22:43:54.597781 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:43:54.597935 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.595076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:38343 (local address 127.11.107.194:43275)
0926 22:43:54.595123 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:54.595145 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:54.595156 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:43:54.595190 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:54.595191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:54.595202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:54.595277 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:54.595450 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.595976 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.597087 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.597540 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.597629 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.597639 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:54.597804 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 22:43:54.597862 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:43:54.597958 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.595016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:43275 (local address 127.11.107.195:38343)
0926 22:43:54.595074 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:54.595087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:54.595132 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:54.595297 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:54.595298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:54.595299 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:54.595379 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.595382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.596050 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.596051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.596978 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.596980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.597628 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.597629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.597722 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.597726 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:43:54.597782 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 22:43:54.597816 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":1}
I20260926 22:43:54.598305 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f925eca7564758a01553255116350a" candidate_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43be633e889e4cf0b844fb1bce8b9384" is_pre_election: true
I20260926 22:43:54.598330 31097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f925eca7564758a01553255116350a" candidate_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d1548184ad5498b8063ff77a25feb8f" is_pre_election: true
I20260926 22:43:54.598460 30963 raft_consensus.cc:2454] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0369c4c5d6bc46a9b628d3ea1b7bcdbd in term 0.
I20260926 22:43:54.598476 31097 raft_consensus.cc:2454] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0369c4c5d6bc46a9b628d3ea1b7bcdbd in term 0.
I20260926 22:43:54.598680 31165 leader_election.cc:302] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0369c4c5d6bc46a9b628d3ea1b7bcdbd, 43be633e889e4cf0b844fb1bce8b9384; no voters:
I20260926 22:43:54.598847 31300 raft_consensus.cc:2781] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:43:54.598891 31300 raft_consensus.cc:482] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:43:54.598906 31300 raft_consensus.cc:3037] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:54.599548 31300 raft_consensus.cc:504] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1548184ad5498b8063ff77a25feb8f" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43275 } } peers { permanent_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38735 } } peers { permanent_uuid: "43be633e889e4cf0b844fb1bce8b9384" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 46621 } }
I20260926 22:43:54.599684 31300 leader_election.cc:288] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [CANDIDATE]: Term 1 election: Requested vote from peers 2d1548184ad5498b8063ff77a25feb8f (127.11.107.194:43275), 43be633e889e4cf0b844fb1bce8b9384 (127.11.107.193:46621)
I20260926 22:43:54.599789 31097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f925eca7564758a01553255116350a" candidate_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d1548184ad5498b8063ff77a25feb8f"
I20260926 22:43:54.599823 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f925eca7564758a01553255116350a" candidate_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43be633e889e4cf0b844fb1bce8b9384"
I20260926 22:43:54.599872 31097 raft_consensus.cc:3037] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:54.599901 30963 raft_consensus.cc:3037] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:54.600778 31097 raft_consensus.cc:2454] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0369c4c5d6bc46a9b628d3ea1b7bcdbd in term 1.
I20260926 22:43:54.600778 30963 raft_consensus.cc:2454] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0369c4c5d6bc46a9b628d3ea1b7bcdbd in term 1.
I20260926 22:43:54.600967 31166 leader_election.cc:302] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0369c4c5d6bc46a9b628d3ea1b7bcdbd, 2d1548184ad5498b8063ff77a25feb8f; no voters:
I20260926 22:43:54.601056 31300 raft_consensus.cc:2781] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:54.601223 31300 raft_consensus.cc:686] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [term 1 LEADER]: Becoming Leader. State: Replica: 0369c4c5d6bc46a9b628d3ea1b7bcdbd, State: Running, Role: LEADER
I20260926 22:43:54.601320 31300 consensus_queue.cc:237] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1548184ad5498b8063ff77a25feb8f" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43275 } } peers { permanent_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38735 } } peers { permanent_uuid: "43be633e889e4cf0b844fb1bce8b9384" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 46621 } }
I20260926 22:43:54.602133 30817 catalog_manager.cc:5495] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0369c4c5d6bc46a9b628d3ea1b7bcdbd (127.11.107.195). New cstate: current_term: 1 leader_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1548184ad5498b8063ff77a25feb8f" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0369c4c5d6bc46a9b628d3ea1b7bcdbd" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 38735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43be633e889e4cf0b844fb1bce8b9384" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 46621 } health_report { overall_health: UNKNOWN } } }
I20260926 22:43:54.613622 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.611318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.11.107.195:38735)
0926 22:43:54.611359 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:54.611361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:54.611369 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:43:54.611421 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:54.611422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:54.611445 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:54.611515 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:43:54.611667 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.612099 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.612862 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.613049 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.613132 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.613309 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:43:54.613394 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:43:54.613482 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:54.613510 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 22:43:54.613528 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:43:54.615211 31097 raft_consensus.cc:1261] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f [term 1 FOLLOWER]: Refusing update from remote peer 0369c4c5d6bc46a9b628d3ea1b7bcdbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:54.615242 30963 raft_consensus.cc:1261] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384 [term 1 FOLLOWER]: Refusing update from remote peer 0369c4c5d6bc46a9b628d3ea1b7bcdbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:54.615490 31300 consensus_queue.cc:1035] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d1548184ad5498b8063ff77a25feb8f" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 43275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:54.615598 31304 consensus_queue.cc:1035] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "43be633e889e4cf0b844fb1bce8b9384" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 46621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:54.617713 31307 mvcc.cc:205] Tried to move back new op lower bound from 7333734951381725184 to 7333734951327641600. Current Snapshot: MvccSnapshot[applied={T|T < 7333734951381725184}]
I20260926 22:43:54.617754 31308 mvcc.cc:205] Tried to move back new op lower bound from 7333734951381725184 to 7333734951327641600. Current Snapshot: MvccSnapshot[applied={T|T < 7333734951381725184}]
I20260926 22:43:54.620582 30818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:54.620657 30818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:54.621346 30818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8bfddeea37644ebfab48f51010983eb7: Sending DeleteTablet for 3 replicas of tablet 32f925eca7564758a01553255116350a
I20260926 22:43:54.621670 31211 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f925eca7564758a01553255116350a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:54 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:37810
I20260926 22:43:54.621737 31077 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f925eca7564758a01553255116350a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:54 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:33684
I20260926 22:43:54.621961 31318 tablet_replica.cc:330] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f: stopping tablet replica
I20260926 22:43:54.621969 30943 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f925eca7564758a01553255116350a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:54 UTC) from {username='kudu', principal='kudu/127.11.107.254@KRBTEST.COM'} at 127.0.0.1:39630
I20260926 22:43:54.622064 31318 raft_consensus.cc:2229] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:43:54.622170 31318 raft_consensus.cc:2258] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:54.622296 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 30880
I20260926 22:43:54.622576 31318 ts_tablet_manager.cc:1905] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:54.622195 31319 tablet_replica.cc:330] T 32f925eca7564758a01553255116350a P 43be633e889e4cf0b844fb1bce8b9384: stopping tablet replica
I20260926 22:43:54.624464 31318 ts_tablet_manager.cc:1918] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:54.624536 31318 log.cc:1196] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/32f925eca7564758a01553255116350a
I20260926 22:43:54.624837 31318 ts_tablet_manager.cc:1939] T 32f925eca7564758a01553255116350a P 2d1548184ad5498b8063ff77a25feb8f: Deleting consensus metadata
I20260926 22:43:54.626097 30804 catalog_manager.cc:4841] TS 2d1548184ad5498b8063ff77a25feb8f (127.11.107.194:43275): tablet 32f925eca7564758a01553255116350a (table test-table [id=1dedeb8c56ac4bbb8680d249e1a8a4d9]) successfully deleted
W20260926 22:43:54.629961 30803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:46621 (error 108)
I20260926 22:43:54.630057 31317 tablet_replica.cc:330] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd: stopping tablet replica
I20260926 22:43:54.630165 31317 raft_consensus.cc:2229] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:43:54.630270 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 31014
I20260926 22:43:54.630295 31317 raft_consensus.cc:2258] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:54.630769 31317 ts_tablet_manager.cc:1905] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:54.631310 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.630516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:46621 (local address 127.0.0.1:39642)
0926 22:43:54.631233 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:54.631277 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":692}
W20260926 22:43:54.631721 30803 catalog_manager.cc:4584] TS 43be633e889e4cf0b844fb1bce8b9384 (127.11.107.193:46621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32f925eca7564758a01553255116350a: Network error: Client connection negotiation failed: client connection to 127.11.107.193:46621: connect: Connection refused (error 111)
I20260926 22:43:54.632723 31317 ts_tablet_manager.cc:1918] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:54.632791 31317 log.cc:1196] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/32f925eca7564758a01553255116350a
I20260926 22:43:54.633044 31317 ts_tablet_manager.cc:1939] T 32f925eca7564758a01553255116350a P 0369c4c5d6bc46a9b628d3ea1b7bcdbd: Deleting consensus metadata
I20260926 22:43:54.633462 30805 catalog_manager.cc:4841] TS 0369c4c5d6bc46a9b628d3ea1b7bcdbd (127.11.107.195:38735): tablet 32f925eca7564758a01553255116350a (table test-table [id=1dedeb8c56ac4bbb8680d249e1a8a4d9]) successfully deleted
I20260926 22:43:54.635943 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 31148
I20260926 22:43:54.640491 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 30786
2026-09-26T22:43:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1987 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T22:43:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:43:54Z Disabled control of system clock
I20260926 22:43:54.667340 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:35801
--webserver_interface=127.11.107.254
--webserver_port=0
--builtin_ntp_servers=127.11.107.212:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:35801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:43:54.747167 31324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:54.747413 31324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:54.747434 31324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:54.748489 31324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:43:54.748521 31324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:54.748534 31324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:54.748546 31324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:43:54.748558 31324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:43:54.749789 31324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:35801
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:35801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:54.750028 31324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:54.750254 31324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:54.754772 31330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:54.754772 31324 server_base.cc:1034] running on GCE node
W20260926 22:43:54.754812 31329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:54.754983 31332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:54.755237 31324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:54.755529 31324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:54.756701 31324 hybrid_clock.cc:648] HybridClock initialized: now 1790462634756693 us; error 34 us; skew 500 ppm
I20260926 22:43:54.757954 31324 webserver.cc:460] Webserver started at http://127.11.107.254:45413/ using document root <none> and password file <none>
I20260926 22:43:54.758203 31324 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:54.758276 31324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:54.758422 31324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:54.759440 31324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "4496858d3828486280c17f698487a378"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "24cfd3307328285b0662424c83fc47a7"
server_key_iv: "65f7a084cf87512312769dad5f151111"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.759792 31324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "4496858d3828486280c17f698487a378"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "24cfd3307328285b0662424c83fc47a7"
server_key_iv: "65f7a084cf87512312769dad5f151111"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.761060 31324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:43:54.761795 31338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:54.762010 31324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:54.762125 31324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "4496858d3828486280c17f698487a378"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "24cfd3307328285b0662424c83fc47a7"
server_key_iv: "65f7a084cf87512312769dad5f151111"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.762197 31324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:54.771386 31324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:54.771647 31324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:54.771785 31324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:54.775473 31324 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:35801
I20260926 22:43:54.775503 31390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:35801 every 8 connection(s)
I20260926 22:43:54.775800 31324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:43:54.776700 31391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:54.778863 31391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378: Bootstrap starting.
I20260926 22:43:54.779347 31391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:54.779569 31391 log.cc:824] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:54.780158 31391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378: No bootstrap required, opened a new log
I20260926 22:43:54.781085 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 31324
I20260926 22:43:54.781208 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:43:54.781385 11695 external_mini_cluster.cc:1428] Setting key 0ee5f91a590202712c486866a9d66d8d
I20260926 22:43:54.782012 31391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4496858d3828486280c17f698487a378" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35801 } }
I20260926 22:43:54.782204 31391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:54.782238 31391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4496858d3828486280c17f698487a378, State: Initialized, Role: FOLLOWER
I20260926 22:43:54.782352 31391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4496858d3828486280c17f698487a378" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35801 } }
I20260926 22:43:54.782424 31391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:43:54.782460 31391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:43:54.782506 31391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:54.783432 31391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4496858d3828486280c17f698487a378" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35801 } }
I20260926 22:43:54.783537 31391 leader_election.cc:302] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4496858d3828486280c17f698487a378; no voters:
I20260926 22:43:54.783682 31391 leader_election.cc:288] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:43:54.783751 31396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:54.783897 31391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:43:54.783913 31396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [term 1 LEADER]: Becoming Leader. State: Replica: 4496858d3828486280c17f698487a378, State: Running, Role: LEADER
I20260926 22:43:54.783998 31396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4496858d3828486280c17f698487a378" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35801 } }
I20260926 22:43:54.784341 31396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4496858d3828486280c17f698487a378. Latest consensus state: current_term: 1 leader_uuid: "4496858d3828486280c17f698487a378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4496858d3828486280c17f698487a378" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35801 } } }
I20260926 22:43:54.784351 31397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4496858d3828486280c17f698487a378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4496858d3828486280c17f698487a378" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35801 } } }
I20260926 22:43:54.784505 31397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:54.784495 31396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:54.785231 31405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:43:54.785951 31405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:43:54.787535 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.781911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.11.107.254:35801)
0926 22:43:54.782145 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:54.782153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:54.782184 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:43:54.782287 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:54.782294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:54.782359 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:54.782501 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:54.782779 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.784216 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.785434 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.785961 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.786091 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.786820 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:54.786823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:54.786824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:54.786988 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:54.787017 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:54.787027 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:54.787098 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:54.787278 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 22:43:54.787362 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
I20260926 22:43:54.787672 31405 catalog_manager.cc:1347] Generated new cluster ID: 3a523db877444aa5a09e54f0e0b9f937
I20260926 22:43:54.787721 31405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:43:54.809525 31405 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:43:54.810328 31405 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:43:54.815311 31405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378: Generated new TSK 0
I20260926 22:43:54.815506 31405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:43:54.817695 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.193:0
--local_ip_for_outbound_sockets=127.11.107.193
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:35801
--builtin_ntp_servers=127.11.107.212:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:43:54.897702 31415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:54.897946 31415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:54.897972 31415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:54.899073 31415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:54.899111 31415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:54.899142 31415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.193
I20260926 22:43:54.900435 31415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.107.193
--webserver_port=0
--tserver_master_addrs=127.11.107.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.193
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:54.900671 31415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:54.900974 31415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:54.906889 31423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:54.906908 31421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:54.907243 31420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:54.907258 31415 server_base.cc:1034] running on GCE node
I20260926 22:43:54.907464 31415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:54.907728 31415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:54.908885 31415 hybrid_clock.cc:648] HybridClock initialized: now 1790462634908866 us; error 36 us; skew 500 ppm
I20260926 22:43:54.910180 31415 webserver.cc:460] Webserver started at http://127.11.107.193:37455/ using document root <none> and password file <none>
I20260926 22:43:54.910421 31415 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:54.910466 31415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:54.910609 31415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:54.911646 31415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/data/instance:
uuid: "7b7f8de032d441b398bb737738af31bf"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "38b52b123aba304024622b563a7a7195"
server_key_iv: "d6406587c50b5c68ffe21f43210276d5"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.911995 31415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance:
uuid: "7b7f8de032d441b398bb737738af31bf"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "38b52b123aba304024622b563a7a7195"
server_key_iv: "d6406587c50b5c68ffe21f43210276d5"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.913312 31415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:43:54.914103 31429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:54.914283 31415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:43:54.914392 31415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/wal
uuid: "7b7f8de032d441b398bb737738af31bf"
format_stamp: "Formatted at 2026-09-26 22:43:54 on dist-test-slave-j8sq"
server_key: "38b52b123aba304024622b563a7a7195"
server_key_iv: "d6406587c50b5c68ffe21f43210276d5"
server_key_version: "encryptionkey@0"
I20260926 22:43:54.914479 31415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:54.927049 31415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:54.927326 31415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:54.927469 31415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:54.927680 31415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:54.927976 31415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:54.928030 31415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:54.928068 31415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:54.928095 31415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:54.933804 31415 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.193:34461
I20260926 22:43:54.933876 31542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.193:34461 every 8 connection(s)
I20260926 22:43:54.934167 31415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/data/info.pb
I20260926 22:43:54.939162 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.934840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:53627 (local address 127.11.107.254:35801)
0926 22:43:54.934888 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:54.934890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:54.935111 (+ 221us) server_negotiation.cc:407] Connection header received
0926 22:43:54.935609 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:54.935611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:54.935633 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:54.935694 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:54.936330 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.936832 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.937803 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.938053 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.938140 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.938891 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:54.938892 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:54.938893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:54.938898 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:54.938911 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:54.938915 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:54.938967 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:54.939056 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 22:43:54.939087 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:43:54.939400 31545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:54.934845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:35801 (local address 127.11.107.193:53627)
0926 22:43:54.934999 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:54.935031 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:54.935508 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:54.935764 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:54.935767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:54.935774 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:54.936240 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.936244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.936940 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.936941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.937698 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:54.937700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:54.938156 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:54.938157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:54.938281 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:54.938757 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:43:54.938758 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:54.938784 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:43:54.938804 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:54.938970 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:54.938975 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:54.938978 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:43:54.939060 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 22:43:54.939167 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260926 22:43:54.939958 31543 heartbeater.cc:347] Connected to a master server at 127.11.107.254:35801
I20260926 22:43:54.940078 31543 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:54.940260 31543 heartbeater.cc:511] Master 127.11.107.254:35801 requested a full tablet report, sending...
I20260926 22:43:54.940833 31355 ts_manager.cc:194] Registered new tserver with Master: 7b7f8de032d441b398bb737738af31bf (127.11.107.193:34461)
I20260926 22:43:54.941471 31355 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.107.193:53627
I20260926 22:43:54.942615 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 31415
I20260926 22:43:54.942700 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-0/wal/instance
I20260926 22:43:54.942879 11695 external_mini_cluster.cc:1428] Setting key 129f013810901a6a0e48017c10505bbf
I20260926 22:43:54.943755 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.194:0
--local_ip_for_outbound_sockets=127.11.107.194
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:35801
--builtin_ntp_servers=127.11.107.212:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:43:55.023285 31546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:55.023512 31546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:55.023550 31546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:55.024690 31546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:55.024753 31546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:55.024796 31546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.194
I20260926 22:43:55.026226 31546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.107.194
--webserver_port=0
--tserver_master_addrs=127.11.107.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.194
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:55.026485 31546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:55.026778 31546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:55.031790 31554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:55.031926 31546 server_base.cc:1034] running on GCE node
W20260926 22:43:55.031793 31551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:55.031967 31552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:55.032275 31546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:55.032531 31546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:55.033723 31546 hybrid_clock.cc:648] HybridClock initialized: now 1790462635033682 us; error 52 us; skew 500 ppm
I20260926 22:43:55.035096 31546 webserver.cc:460] Webserver started at http://127.11.107.194:42699/ using document root <none> and password file <none>
I20260926 22:43:55.035375 31546 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:55.035449 31546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:55.035629 31546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:55.036720 31546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/data/instance:
uuid: "5ffd423eff9145c9a5e263fa5ae0aec7"
format_stamp: "Formatted at 2026-09-26 22:43:55 on dist-test-slave-j8sq"
server_key: "2c9c46111fe2f9501644c98b13d87130"
server_key_iv: "1394c8b55c56e1a5b7afd160994a8672"
server_key_version: "encryptionkey@0"
I20260926 22:43:55.037086 31546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance:
uuid: "5ffd423eff9145c9a5e263fa5ae0aec7"
format_stamp: "Formatted at 2026-09-26 22:43:55 on dist-test-slave-j8sq"
server_key: "2c9c46111fe2f9501644c98b13d87130"
server_key_iv: "1394c8b55c56e1a5b7afd160994a8672"
server_key_version: "encryptionkey@0"
I20260926 22:43:55.038470 31546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 22:43:55.039311 31560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:55.039486 31546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:55.039613 31546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/wal
uuid: "5ffd423eff9145c9a5e263fa5ae0aec7"
format_stamp: "Formatted at 2026-09-26 22:43:55 on dist-test-slave-j8sq"
server_key: "2c9c46111fe2f9501644c98b13d87130"
server_key_iv: "1394c8b55c56e1a5b7afd160994a8672"
server_key_version: "encryptionkey@0"
I20260926 22:43:55.039705 31546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:55.060282 31546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:55.060621 31546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:55.060782 31546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:55.061017 31546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:55.061347 31546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:55.061401 31546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:55.061444 31546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:55.061472 31546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:55.067440 31546 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.194:39173
I20260926 22:43:55.067485 31673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.194:39173 every 8 connection(s)
I20260926 22:43:55.067850 31546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/data/info.pb
I20260926 22:43:55.068315 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 31546
I20260926 22:43:55.068435 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/wal/instance
I20260926 22:43:55.068667 11695 external_mini_cluster.cc:1428] Setting key 06b66c3b35c8d37a3c6ee3a139f25b1a
I20260926 22:43:55.069733 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.107.195:0
--local_ip_for_outbound_sockets=127.11.107.195
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:35801
--builtin_ntp_servers=127.11.107.212:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 22:43:55.074358 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.068665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.194:55439 (local address 127.11.107.254:35801)
0926 22:43:55.068721 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:55.068723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:55.069081 (+ 358us) server_negotiation.cc:407] Connection header received
0926 22:43:55.069742 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:55.069743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:55.069753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:55.069793 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:55.070873 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.071360 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.072895 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.073080 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.073141 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.074087 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:55.074088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:55.074088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:55.074092 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:55.074105 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:55.074109 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:55.074167 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:55.074279 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 22:43:55.074302 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 22:43:55.074715 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.068708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:35801 (local address 127.11.107.194:55439)
0926 22:43:55.068920 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:55.068985 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:55.069632 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:55.070034 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:55.070040 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:55.070050 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:55.070763 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.070771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.071460 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.071462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.072804 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.072807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.073171 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.073172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.073337 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.073921 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:43:55.073922 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:55.073962 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:43:55.073994 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:55.074192 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:55.074197 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:55.074201 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:43:55.074280 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 22:43:55.074397 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
I20260926 22:43:55.075255 31674 heartbeater.cc:347] Connected to a master server at 127.11.107.254:35801
I20260926 22:43:55.075343 31674 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:55.075529 31674 heartbeater.cc:511] Master 127.11.107.254:35801 requested a full tablet report, sending...
I20260926 22:43:55.075904 31355 ts_manager.cc:194] Registered new tserver with Master: 5ffd423eff9145c9a5e263fa5ae0aec7 (127.11.107.194:39173)
I20260926 22:43:55.076453 31355 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.107.194:55439
W20260926 22:43:55.150254 31677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:55.150465 31677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:55.150501 31677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:55.151718 31677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:55.151772 31677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:55.151813 31677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.107.195
I20260926 22:43:55.153136 31677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.107.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.107.195
--webserver_port=0
--tserver_master_addrs=127.11.107.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.107.195
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:55.153388 31677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:55.153666 31677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:55.158394 31683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:55.158383 31685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:55.158383 31682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:55.158515 31677 server_base.cc:1034] running on GCE node
I20260926 22:43:55.158857 31677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:55.159057 31677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:55.160199 31677 hybrid_clock.cc:648] HybridClock initialized: now 1790462635160190 us; error 27 us; skew 500 ppm
I20260926 22:43:55.161434 31677 webserver.cc:460] Webserver started at http://127.11.107.195:43551/ using document root <none> and password file <none>
I20260926 22:43:55.161620 31677 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:55.161669 31677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:55.161753 31677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:55.162792 31677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/data/instance:
uuid: "7acbc6e4ccc74281aac0157ee3d3d049"
format_stamp: "Formatted at 2026-09-26 22:43:55 on dist-test-slave-j8sq"
server_key: "6e32e2c19625a86f05dc95f52bcfb7d9"
server_key_iv: "fd934d585ff8bf26c039601a817eb73a"
server_key_version: "encryptionkey@0"
I20260926 22:43:55.163074 31677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance:
uuid: "7acbc6e4ccc74281aac0157ee3d3d049"
format_stamp: "Formatted at 2026-09-26 22:43:55 on dist-test-slave-j8sq"
server_key: "6e32e2c19625a86f05dc95f52bcfb7d9"
server_key_iv: "fd934d585ff8bf26c039601a817eb73a"
server_key_version: "encryptionkey@0"
I20260926 22:43:55.164304 31677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:43:55.165151 31691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:55.165329 31677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:43:55.165427 31677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/wal
uuid: "7acbc6e4ccc74281aac0157ee3d3d049"
format_stamp: "Formatted at 2026-09-26 22:43:55 on dist-test-slave-j8sq"
server_key: "6e32e2c19625a86f05dc95f52bcfb7d9"
server_key_iv: "fd934d585ff8bf26c039601a817eb73a"
server_key_version: "encryptionkey@0"
I20260926 22:43:55.165506 31677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:55.186465 31677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:55.186823 31677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:55.186965 31677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:55.187201 31677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:43:55.187530 31677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:43:55.187582 31677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:55.187618 31677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:43:55.187646 31677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:55.193089 31677 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.195:39049
I20260926 22:43:55.193137 31804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.195:39049 every 8 connection(s)
I20260926 22:43:55.193495 31677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/data/info.pb
I20260926 22:43:55.194948 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 31677
I20260926 22:43:55.195083 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/wal/instance
I20260926 22:43:55.195277 11695 external_mini_cluster.cc:1428] Setting key 4418c8ebbc0f82452ff6bfdf01e59df3
I20260926 22:43:55.198735 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.194270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.195:51535 (local address 127.11.107.254:35801)
0926 22:43:55.194317 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:55.194319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:55.194563 (+ 244us) server_negotiation.cc:407] Connection header received
0926 22:43:55.195021 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:55.195022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:55.195035 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:55.195093 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:55.195965 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.196570 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.197488 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.197668 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.197718 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.198508 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:55.198509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:55.198509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:55.198513 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:55.198525 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:55.198529 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:55.198571 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:55.198649 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 22:43:55.198671 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:43:55.198947 31807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.194231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:35801 (local address 127.11.107.195:51535)
0926 22:43:55.194406 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:55.194446 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:55.194914 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:55.195163 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:55.195168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:55.195178 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:43:55.195836 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.195845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.196654 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.196655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.197414 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.197416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.197736 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.197737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.197893 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.198384 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:43:55.198384 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:43:55.198411 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:43:55.198432 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:43:55.198580 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:43:55.198584 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:43:55.198587 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:43:55.198647 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 22:43:55.198718 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260926 22:43:55.199337 31805 heartbeater.cc:347] Connected to a master server at 127.11.107.254:35801
I20260926 22:43:55.199414 31805 heartbeater.cc:464] Registering TS with master...
I20260926 22:43:55.199601 31805 heartbeater.cc:511] Master 127.11.107.254:35801 requested a full tablet report, sending...
I20260926 22:43:55.199978 31355 ts_manager.cc:194] Registered new tserver with Master: 7acbc6e4ccc74281aac0157ee3d3d049 (127.11.107.195:39049)
I20260926 22:43:55.200348 31355 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.107.195:51535
I20260926 22:43:55.206861 11695 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:43:55.211069 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.208164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47382 (local address 127.11.107.254:35801)
0926 22:43:55.208261 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:55.208263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:55.208362 (+ 99us) server_negotiation.cc:407] Connection header received
0926 22:43:55.208457 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:55.208458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:55.208469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:55.208520 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:55.208684 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.209138 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.209936 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.210135 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.210225 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.210829 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:55.210831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:55.210831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:55.210835 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:55.210847 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:55.210851 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:55.210893 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:55.210965 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 22:43:55.210987 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 22:43:55.212525 31355 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:43:55.214010 31355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:43:55.220403 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.216588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.11.107.195:39049)
0926 22:43:55.216958 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:55.216965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:55.216975 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:43:55.217013 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:55.217015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:55.217029 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:55.217104 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:55.217222 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.218229 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.219263 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.219987 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.220068 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.220155 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:55.220286 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 22:43:55.220321 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260926 22:43:55.220435 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.216097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:39173 (local address 127.0.0.1:50194)
0926 22:43:55.216221 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:55.216259 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:55.216322 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:55.216621 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:55.216625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:55.216626 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:55.216688 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.216689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.218031 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.218031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.219319 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.219322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.220163 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.220163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.220213 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.220216 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:43:55.220266 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 22:43:55.220327 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":76,"thread_start_us":82,"threads_started":1}
I20260926 22:43:55.220434 31816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.216319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:39049 (local address 127.0.0.1:57950)
0926 22:43:55.216858 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:55.216868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:55.216910 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:55.217102 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:55.217102 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:55.217104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:55.217137 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.217139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.218311 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.218312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.219180 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.219183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.220074 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.220075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.220170 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.220186 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 22:43:55.220261 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 22:43:55.220326 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":69,"threads_started":1}
I20260926 22:43:55.220963 31815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.216231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.11.107.194:39173)
0926 22:43:55.216461 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:55.216465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:55.216474 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:43:55.216513 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:55.216515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:55.216532 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:55.216617 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:55.216807 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.217964 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.219402 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.220085 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.220148 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.220220 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:55.220836 (+ 616us) server_negotiation.cc:299] Negotiation successful
0926 22:43:55.220881 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260926 22:43:55.221388 31739 tablet_service.cc:1467] Processing CreateTablet for tablet 96c649ad25a1428b9099bb320e160d18 (DEFAULT_TABLE table=test-table [id=bc8ab4b2fc334aa9aa51760de87bd468]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:55.221730 31739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c649ad25a1428b9099bb320e160d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:55.221716 31608 tablet_service.cc:1467] Processing CreateTablet for tablet 96c649ad25a1428b9099bb320e160d18 (DEFAULT_TABLE table=test-table [id=bc8ab4b2fc334aa9aa51760de87bd468]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:55.222054 31608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c649ad25a1428b9099bb320e160d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:55.224388 31819 tablet_bootstrap.cc:492] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049: Bootstrap starting.
I20260926 22:43:55.224725 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.216110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:34461 (local address 127.0.0.1:59582)
0926 22:43:55.216442 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:55.216452 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:55.216502 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:55.216732 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:55.216733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:55.216735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:55.216780 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.216781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.217986 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.217987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.219320 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.219322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.224485 (+ 5163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.224486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.224550 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.224555 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:43:55.224589 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 22:43:55.224632 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":33,"thread_start_us":92,"threads_started":1}
I20260926 22:43:55.225039 31819 tablet_bootstrap.cc:654] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:55.225121 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.216412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.11.107.193:34461)
0926 22:43:55.216588 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:55.216593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:55.216602 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:43:55.216638 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:55.216640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:55.216653 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:55.216721 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:55.216892 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.217911 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.220275 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.220940 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.224864 (+ 3924us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.224966 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:55.224992 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 22:43:55.225034 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260926 22:43:55.225345 31819 log.cc:824] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:55.225487 31820 tablet_bootstrap.cc:492] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7: Bootstrap starting.
I20260926 22:43:55.225734 31477 tablet_service.cc:1467] Processing CreateTablet for tablet 96c649ad25a1428b9099bb320e160d18 (DEFAULT_TABLE table=test-table [id=bc8ab4b2fc334aa9aa51760de87bd468]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:43:55.226095 31477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c649ad25a1428b9099bb320e160d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:55.226115 31819 tablet_bootstrap.cc:492] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049: No bootstrap required, opened a new log
I20260926 22:43:55.226137 31820 tablet_bootstrap.cc:654] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:55.226191 31819 ts_tablet_manager.cc:1397] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:43:55.226426 31820 log.cc:824] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:55.227188 31820 tablet_bootstrap.cc:492] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7: No bootstrap required, opened a new log
I20260926 22:43:55.227263 31820 ts_tablet_manager.cc:1397] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:43:55.228320 31819 raft_consensus.cc:348] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffd423eff9145c9a5e263fa5ae0aec7" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39173 } } peers { permanent_uuid: "7b7f8de032d441b398bb737738af31bf" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34461 } } peers { permanent_uuid: "7acbc6e4ccc74281aac0157ee3d3d049" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39049 } }
I20260926 22:43:55.228466 31819 raft_consensus.cc:374] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:55.228506 31819 raft_consensus.cc:729] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7acbc6e4ccc74281aac0157ee3d3d049, State: Initialized, Role: FOLLOWER
I20260926 22:43:55.228642 31819 consensus_queue.cc:260] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffd423eff9145c9a5e263fa5ae0aec7" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39173 } } peers { permanent_uuid: "7b7f8de032d441b398bb737738af31bf" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34461 } } peers { permanent_uuid: "7acbc6e4ccc74281aac0157ee3d3d049" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39049 } }
I20260926 22:43:55.228883 31819 ts_tablet_manager.cc:1428] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 22:43:55.229012 31805 heartbeater.cc:503] Master 127.11.107.254:35801 was elected leader, sending a full tablet report...
I20260926 22:43:55.229746 31824 tablet_bootstrap.cc:492] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf: Bootstrap starting.
I20260926 22:43:55.229825 31820 raft_consensus.cc:348] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffd423eff9145c9a5e263fa5ae0aec7" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39173 } } peers { permanent_uuid: "7b7f8de032d441b398bb737738af31bf" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34461 } } peers { permanent_uuid: "7acbc6e4ccc74281aac0157ee3d3d049" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39049 } }
I20260926 22:43:55.230000 31820 raft_consensus.cc:374] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:55.230050 31820 raft_consensus.cc:729] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ffd423eff9145c9a5e263fa5ae0aec7, State: Initialized, Role: FOLLOWER
I20260926 22:43:55.230192 31820 consensus_queue.cc:260] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffd423eff9145c9a5e263fa5ae0aec7" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39173 } } peers { permanent_uuid: "7b7f8de032d441b398bb737738af31bf" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34461 } } peers { permanent_uuid: "7acbc6e4ccc74281aac0157ee3d3d049" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39049 } }
I20260926 22:43:55.230427 31824 tablet_bootstrap.cc:654] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:55.230428 31820 ts_tablet_manager.cc:1428] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 22:43:55.230473 31674 heartbeater.cc:503] Master 127.11.107.254:35801 was elected leader, sending a full tablet report...
I20260926 22:43:55.230711 31824 log.cc:824] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:55.231467 31824 tablet_bootstrap.cc:492] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf: No bootstrap required, opened a new log
I20260926 22:43:55.231518 31824 ts_tablet_manager.cc:1397] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:43:55.233168 31824 raft_consensus.cc:348] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffd423eff9145c9a5e263fa5ae0aec7" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39173 } } peers { permanent_uuid: "7b7f8de032d441b398bb737738af31bf" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34461 } } peers { permanent_uuid: "7acbc6e4ccc74281aac0157ee3d3d049" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39049 } }
I20260926 22:43:55.233289 31824 raft_consensus.cc:374] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:55.233309 31824 raft_consensus.cc:729] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7f8de032d441b398bb737738af31bf, State: Initialized, Role: FOLLOWER
I20260926 22:43:55.233439 31824 consensus_queue.cc:260] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffd423eff9145c9a5e263fa5ae0aec7" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39173 } } peers { permanent_uuid: "7b7f8de032d441b398bb737738af31bf" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34461 } } peers { permanent_uuid: "7acbc6e4ccc74281aac0157ee3d3d049" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39049 } }
I20260926 22:43:55.233682 31824 ts_tablet_manager.cc:1428] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:43:55.233731 31543 heartbeater.cc:503] Master 127.11.107.254:35801 was elected leader, sending a full tablet report...
I20260926 22:43:55.264780 31827 raft_consensus.cc:482] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:43:55.264909 31827 raft_consensus.cc:504] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffd423eff9145c9a5e263fa5ae0aec7" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39173 } } peers { permanent_uuid: "7b7f8de032d441b398bb737738af31bf" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34461 } } peers { permanent_uuid: "7acbc6e4ccc74281aac0157ee3d3d049" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39049 } }
I20260926 22:43:55.265369 31827 leader_election.cc:288] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ffd423eff9145c9a5e263fa5ae0aec7 (127.11.107.194:39173), 7acbc6e4ccc74281aac0157ee3d3d049 (127.11.107.195:39049)
I20260926 22:43:55.268533 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.265624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:34113 (local address 127.11.107.195:39049)
0926 22:43:55.265693 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:55.265695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:55.265705 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:43:55.265744 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:55.265745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:55.265754 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:55.265835 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:55.266067 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.266619 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.267592 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.268183 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.268253 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.268261 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:55.268429 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 22:43:55.268470 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:43:55.268533 31545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.265519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.195:39049 (local address 127.11.107.193:34113)
0926 22:43:55.265553 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:55.265579 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:55.265622 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:55.265878 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:55.265879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:55.265882 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:55.265985 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.265987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.266700 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.266701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.267499 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.267502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.268262 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.268263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.268320 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.268323 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:43:55.268418 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 22:43:55.268459 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 22:43:55.268736 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.265519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.194:39173 (local address 127.11.107.193:44635)
0926 22:43:55.265891 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:55.265900 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:43:55.265956 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:43:55.266113 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:43:55.266113 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:43:55.266115 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:43:55.266180 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.266182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.266860 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.266861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.267613 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:43:55.267615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.268350 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.268351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:43:55.268506 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.268510 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:43:55.268612 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 22:43:55.268651 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":183,"thread_start_us":122,"threads_started":1}
I20260926 22:43:55.268738 31815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.265655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.107.193:44635 (local address 127.11.107.194:39173)
0926 22:43:55.265705 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:55.265708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:55.265993 (+ 285us) server_negotiation.cc:407] Connection header received
0926 22:43:55.266028 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:55.266029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:55.266041 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:55.266107 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:43:55.266249 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.266770 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.267748 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.268285 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.268326 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.268335 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:55.268620 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 22:43:55.268661 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:43:55.268937 31759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c649ad25a1428b9099bb320e160d18" candidate_uuid: "7b7f8de032d441b398bb737738af31bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acbc6e4ccc74281aac0157ee3d3d049" is_pre_election: true
I20260926 22:43:55.269140 31759 raft_consensus.cc:2454] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7f8de032d441b398bb737738af31bf in term 0.
I20260926 22:43:55.269083 31628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c649ad25a1428b9099bb320e160d18" candidate_uuid: "7b7f8de032d441b398bb737738af31bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ffd423eff9145c9a5e263fa5ae0aec7" is_pre_election: true
I20260926 22:43:55.269230 31628 raft_consensus.cc:2454] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7f8de032d441b398bb737738af31bf in term 0.
I20260926 22:43:55.269387 31432 leader_election.cc:302] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7acbc6e4ccc74281aac0157ee3d3d049, 7b7f8de032d441b398bb737738af31bf; no voters:
I20260926 22:43:55.269639 31827 raft_consensus.cc:2781] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:43:55.269709 31827 raft_consensus.cc:482] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:43:55.269729 31827 raft_consensus.cc:3037] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:55.270529 31827 raft_consensus.cc:504] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffd423eff9145c9a5e263fa5ae0aec7" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39173 } } peers { permanent_uuid: "7b7f8de032d441b398bb737738af31bf" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34461 } } peers { permanent_uuid: "7acbc6e4ccc74281aac0157ee3d3d049" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39049 } }
I20260926 22:43:55.270715 31827 leader_election.cc:288] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [CANDIDATE]: Term 1 election: Requested vote from peers 5ffd423eff9145c9a5e263fa5ae0aec7 (127.11.107.194:39173), 7acbc6e4ccc74281aac0157ee3d3d049 (127.11.107.195:39049)
I20260926 22:43:55.270855 31628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c649ad25a1428b9099bb320e160d18" candidate_uuid: "7b7f8de032d441b398bb737738af31bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ffd423eff9145c9a5e263fa5ae0aec7"
I20260926 22:43:55.270923 31759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c649ad25a1428b9099bb320e160d18" candidate_uuid: "7b7f8de032d441b398bb737738af31bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acbc6e4ccc74281aac0157ee3d3d049"
I20260926 22:43:55.270959 31628 raft_consensus.cc:3037] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:55.271005 31759 raft_consensus.cc:3037] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:55.271720 31759 raft_consensus.cc:2454] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7f8de032d441b398bb737738af31bf in term 1.
I20260926 22:43:55.271725 31628 raft_consensus.cc:2454] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7f8de032d441b398bb737738af31bf in term 1.
I20260926 22:43:55.271921 31433 leader_election.cc:302] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ffd423eff9145c9a5e263fa5ae0aec7, 7b7f8de032d441b398bb737738af31bf; no voters:
I20260926 22:43:55.272056 31827 raft_consensus.cc:2781] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:55.272258 31827 raft_consensus.cc:686] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [term 1 LEADER]: Becoming Leader. State: Replica: 7b7f8de032d441b398bb737738af31bf, State: Running, Role: LEADER
I20260926 22:43:55.272380 31827 consensus_queue.cc:237] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffd423eff9145c9a5e263fa5ae0aec7" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39173 } } peers { permanent_uuid: "7b7f8de032d441b398bb737738af31bf" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34461 } } peers { permanent_uuid: "7acbc6e4ccc74281aac0157ee3d3d049" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39049 } }
I20260926 22:43:55.273289 31355 catalog_manager.cc:5495] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b7f8de032d441b398bb737738af31bf (127.11.107.193). New cstate: current_term: 1 leader_uuid: "7b7f8de032d441b398bb737738af31bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ffd423eff9145c9a5e263fa5ae0aec7" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b7f8de032d441b398bb737738af31bf" member_type: VOTER last_known_addr { host: "127.11.107.193" port: 34461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7acbc6e4ccc74281aac0157ee3d3d049" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39049 } health_report { overall_health: UNKNOWN } } }
I20260926 22:43:55.279428 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.277095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.11.107.193:34461)
0926 22:43:55.277140 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:55.277142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:55.277157 (+ 15us) server_negotiation.cc:407] Connection header received
0926 22:43:55.277188 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:55.277189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:55.277207 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:55.277276 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:43:55.277398 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.277906 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.278665 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:55.278865 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:55.278953 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:55.279120 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:43:55.279216 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:43:55.279305 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:55.279324 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 22:43:55.279342 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:43:55.280920 31628 raft_consensus.cc:1261] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7 [term 1 FOLLOWER]: Refusing update from remote peer 7b7f8de032d441b398bb737738af31bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:55.280925 31759 raft_consensus.cc:1261] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049 [term 1 FOLLOWER]: Refusing update from remote peer 7b7f8de032d441b398bb737738af31bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:43:55.281208 31827 consensus_queue.cc:1035] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7acbc6e4ccc74281aac0157ee3d3d049" member_type: VOTER last_known_addr { host: "127.11.107.195" port: 39049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:55.281329 31829 consensus_queue.cc:1035] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ffd423eff9145c9a5e263fa5ae0aec7" member_type: VOTER last_known_addr { host: "127.11.107.194" port: 39173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:43:55.283561 31832 mvcc.cc:205] Tried to move back new op lower bound from 7333734954108506112 to 7333734954076561408. Current Snapshot: MvccSnapshot[applied={T|T < 7333734954108506112}]
I20260926 22:43:55.283748 31833 mvcc.cc:205] Tried to move back new op lower bound from 7333734954108506112 to 7333734954076561408. Current Snapshot: MvccSnapshot[applied={T|T < 7333734954108506112}]
I20260926 22:43:55.283890 31834 mvcc.cc:205] Tried to move back new op lower bound from 7333734954108506112 to 7333734954076561408. Current Snapshot: MvccSnapshot[applied={T|T < 7333734954108506112}]
I20260926 22:43:55.286453 31355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:55.286523 31355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:43:55.287201 31355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4496858d3828486280c17f698487a378: Sending DeleteTablet for 3 replicas of tablet 96c649ad25a1428b9099bb320e160d18
I20260926 22:43:55.287503 31477 tablet_service.cc:1514] Processing DeleteTablet for tablet 96c649ad25a1428b9099bb320e160d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:55 UTC) from {username='slave'} at 127.0.0.1:59582
I20260926 22:43:55.287549 31739 tablet_service.cc:1514] Processing DeleteTablet for tablet 96c649ad25a1428b9099bb320e160d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:55 UTC) from {username='slave'} at 127.0.0.1:57950
I20260926 22:43:55.287778 31842 tablet_replica.cc:330] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf: stopping tablet replica
I20260926 22:43:55.287782 31608 tablet_service.cc:1514] Processing DeleteTablet for tablet 96c649ad25a1428b9099bb320e160d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:43:55 UTC) from {username='slave'} at 127.0.0.1:50194
I20260926 22:43:55.287870 31842 raft_consensus.cc:2229] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:43:55.287982 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 31415
I20260926 22:43:55.288044 31842 raft_consensus.cc:2258] T 96c649ad25a1428b9099bb320e160d18 P 7b7f8de032d441b398bb737738af31bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:55.289964 31844 tablet_replica.cc:330] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7: stopping tablet replica
I20260926 22:43:55.290073 31844 raft_consensus.cc:2229] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:43:55.290185 31844 raft_consensus.cc:2258] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:55.290583 31844 ts_tablet_manager.cc:1905] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:55.292563 31844 ts_tablet_manager.cc:1918] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:55.292639 31844 log.cc:1196] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-1/wal/wals/96c649ad25a1428b9099bb320e160d18
I20260926 22:43:55.292872 31844 ts_tablet_manager.cc:1939] T 96c649ad25a1428b9099bb320e160d18 P 5ffd423eff9145c9a5e263fa5ae0aec7: Deleting consensus metadata
I20260926 22:43:55.293534 31342 catalog_manager.cc:4841] TS 5ffd423eff9145c9a5e263fa5ae0aec7 (127.11.107.194:39173): tablet 96c649ad25a1428b9099bb320e160d18 (table test-table [id=bc8ab4b2fc334aa9aa51760de87bd468]) successfully deleted
I20260926 22:43:55.293977 31843 tablet_replica.cc:330] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049: stopping tablet replica
I20260926 22:43:55.294085 31843 raft_consensus.cc:2229] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 22:43:55.294296 31341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.107.193:34461 (error 108)
I20260926 22:43:55.294324 31843 raft_consensus.cc:2258] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:43:55.294533 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 31546
I20260926 22:43:55.296183 31843 ts_tablet_manager.cc:1905] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:43:55.298125 31843 ts_tablet_manager.cc:1918] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:43:55.298198 31843 log.cc:1196] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049: Deleting WAL directory at /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462400949426-11695-0/minicluster-data/ts-2/wal/wals/96c649ad25a1428b9099bb320e160d18
I20260926 22:43:55.298257 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:55.298113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.193:34461 (local address 127.0.0.1:59612)
0926 22:43:55.298168 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 22:43:55.298214 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.193:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 22:43:55.298394 31341 catalog_manager.cc:4584] TS 7b7f8de032d441b398bb737738af31bf (127.11.107.193:34461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96c649ad25a1428b9099bb320e160d18: Network error: Client connection negotiation failed: client connection to 127.11.107.193:34461: connect: Connection refused (error 111)
I20260926 22:43:55.298448 31843 ts_tablet_manager.cc:1939] T 96c649ad25a1428b9099bb320e160d18 P 7acbc6e4ccc74281aac0157ee3d3d049: Deleting consensus metadata
I20260926 22:43:55.298827 31341 catalog_manager.cc:4841] TS 7acbc6e4ccc74281aac0157ee3d3d049 (127.11.107.195:39049): tablet 96c649ad25a1428b9099bb320e160d18 (table test-table [id=bc8ab4b2fc334aa9aa51760de87bd468]) successfully deleted
I20260926 22:43:55.300019 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 31677
I20260926 22:43:55.304870 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 31324
2026-09-26T22:43:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (663 ms)
[----------] 6 tests from EncryptionPolicyTest (8491 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:43:55 dist-test-slave-j8sq krb5kdc[31848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:43:55 dist-test-slave-j8sq krb5kdc[31848](info): set up 2 sockets
Sep 26 22:43:55 dist-test-slave-j8sq krb5kdc[31848](info): commencing operation
krb5kdc: starting...
W20260926 22:43:56.334905 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:43:56 dist-test-slave-j8sq krb5kdc[31848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:43:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:43:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:43:56.450726 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:56.528824 31861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:56.529069 31861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:56.529109 31861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:56.530285 31861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:43:56.530359 31861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:56.530385 31861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:56.530414 31861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:43:56.530443 31861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:43:56.531947 31861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:56.532220 31861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:56.532459 31861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:56.537411 31866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:56.537412 31869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:56.537421 31867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:56.537447 31861 server_base.cc:1034] running on GCE node
I20260926 22:43:56.537907 31861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:56.538239 31861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:56.539415 31861 hybrid_clock.cc:648] HybridClock initialized: now 1790462636539400 us; error 26 us; skew 500 ppm
Sep 26 22:43:56 dist-test-slave-j8sq krb5kdc[31848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:56.542605 31861 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:43:56.543059 31861 webserver.cc:460] Webserver started at http://127.0.0.1:43411/ using document root <none> and password file <none>
I20260926 22:43:56.543315 31861 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:56.543362 31861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:56.543453 31861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:56.544519 31861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "9d8e893ac00a4935955d88c3dbb5c27d"
format_stamp: "Formatted at 2026-09-26 22:43:56 on dist-test-slave-j8sq"
server_key: "1328b674e7dde2517081ccb2fc35ead0"
server_key_iv: "01a7e41dbcb1680c1a1507312132f53d"
server_key_version: "encryptionkey@0"
I20260926 22:43:56.544826 31861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "9d8e893ac00a4935955d88c3dbb5c27d"
format_stamp: "Formatted at 2026-09-26 22:43:56 on dist-test-slave-j8sq"
server_key: "1328b674e7dde2517081ccb2fc35ead0"
server_key_iv: "01a7e41dbcb1680c1a1507312132f53d"
server_key_version: "encryptionkey@0"
I20260926 22:43:56.546139 31861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:43:56.546815 31876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:56.547021 31861 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:43:56.547111 31861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "9d8e893ac00a4935955d88c3dbb5c27d"
format_stamp: "Formatted at 2026-09-26 22:43:56 on dist-test-slave-j8sq"
server_key: "1328b674e7dde2517081ccb2fc35ead0"
server_key_iv: "01a7e41dbcb1680c1a1507312132f53d"
server_key_version: "encryptionkey@0"
I20260926 22:43:56.547169 31861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:56.586303 31861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:56.588750 31861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:56.588867 31861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:56.592880 31861 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36143
I20260926 22:43:56.592963 31928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36143 every 8 connection(s)
I20260926 22:43:56.593336 31861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:43:56.594235 31929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:56.594990 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 31861
I20260926 22:43:56.595101 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:43:56.595299 11695 external_mini_cluster.cc:1428] Setting key 39029c5ecdf7c87b5aabe698d61fc0fa
I20260926 22:43:56.597522 31929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d: Bootstrap starting.
I20260926 22:43:56.598162 31929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:56.598461 31929 log.cc:824] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:56.599092 31929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d: No bootstrap required, opened a new log
I20260926 22:43:56.600842 31929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8e893ac00a4935955d88c3dbb5c27d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260926 22:43:56.600991 31929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:56.601037 31929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8e893ac00a4935955d88c3dbb5c27d, State: Initialized, Role: FOLLOWER
I20260926 22:43:56.601188 31929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8e893ac00a4935955d88c3dbb5c27d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260926 22:43:56.601280 31929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:43:56.601322 31929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:43:56.601370 31929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:56.601974 31929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8e893ac00a4935955d88c3dbb5c27d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260926 22:43:56.602077 31929 leader_election.cc:302] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d8e893ac00a4935955d88c3dbb5c27d; no voters:
I20260926 22:43:56.602239 31929 leader_election.cc:288] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:43:56.602336 31934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [term 1 FOLLOWER]: Leader election won for term 1
Sep 26 22:43:56 dist-test-slave-j8sq krb5kdc[31848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 22:43:56.602507 31929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:43:56.602535 31934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [term 1 LEADER]: Becoming Leader. State: Replica: 9d8e893ac00a4935955d88c3dbb5c27d, State: Running, Role: LEADER
I20260926 22:43:56.602628 31934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8e893ac00a4935955d88c3dbb5c27d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260926 22:43:56.602952 31934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d8e893ac00a4935955d88c3dbb5c27d. Latest consensus state: current_term: 1 leader_uuid: "9d8e893ac00a4935955d88c3dbb5c27d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8e893ac00a4935955d88c3dbb5c27d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } } }
I20260926 22:43:56.603032 31934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:56.603355 31942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:43:56.603364 31935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d8e893ac00a4935955d88c3dbb5c27d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8e893ac00a4935955d88c3dbb5c27d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } } }
I20260926 22:43:56.603505 31935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:56.604005 31942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:43:56.605652 31942 catalog_manager.cc:1347] Generated new cluster ID: 1cd49260d9ea4ea49005e277a241354f
I20260926 22:43:56.605705 31942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:43:56.608778 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:56.595847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32803 (local address 127.0.0.1:36143)
0926 22:43:56.596024 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:56.596033 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:56.596068 (+ 35us) server_negotiation.cc:407] Connection header received
0926 22:43:56.596537 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:56.596543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:56.596589 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:56.596711 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:56.597503 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:56.598685 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:56.599551 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:56.600024 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:56.600158 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:56.603000 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:56.603004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:56.603005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:56.603016 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:56.607022 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:56.607537 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:56.607540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:56.607540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:56.607647 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:56.607909 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:56.607909 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:56.607910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:56.608176 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:56.608324 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:56.608480 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 22:43:56.608573 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260926 22:43:56.609578 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:43:56.611610 31942 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:43:56.612408 31942 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:43:56.617197 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:56.610952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56217 (local address 127.0.0.1:36143)
0926 22:43:56.611008 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:56.611010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:56.611415 (+ 405us) server_negotiation.cc:407] Connection header received
0926 22:43:56.611449 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:56.611450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:56.611461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:56.611531 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:56.612067 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:56.612676 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:56.613378 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:56.613580 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:56.613669 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:56.614766 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:56.614767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:56.614768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:43:56.614774 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:56.616082 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:56.616435 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:56.616436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:56.616436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:56.616492 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:43:56.616705 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:43:56.616706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:43:56.616706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:43:56.616804 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:56.616925 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:56.617039 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 22:43:56.617098 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":26}
I20260926 22:43:56.617962 31942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d8e893ac00a4935955d88c3dbb5c27d: Generated new TSK 0
I20260926 22:43:56.618124 31942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:43:56.622457 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 31861
W20260926 22:43:56.626933 31846 connection.cc:535] client connection to 127.0.0.1:36143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36143): Connection reset by peer (error 104)
2026-09-26T22:43:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:43:56 dist-test-slave-j8sq krb5kdc[31960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:43:56 dist-test-slave-j8sq krb5kdc[31960](info): set up 2 sockets
Sep 26 22:43:56 dist-test-slave-j8sq krb5kdc[31960](info): commencing operation
krb5kdc: starting...
W20260926 22:43:57.691354 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:43:57 dist-test-slave-j8sq krb5kdc[31960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:43:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:43:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:43:57.806908 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:57.886379 31973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:57.886596 31973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:57.886631 31973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:57.887751 31973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:43:57.887816 31973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:57.887842 31973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:57.887874 31973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:43:57.887897 31973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:43:57.889364 31973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:57.889622 31973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:57.889832 31973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:43:57.894613 31973 server_base.cc:1034] running on GCE node
W20260926 22:43:57.894582 31979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:57.894608 31978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:57.894569 31981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:57.895013 31973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:57.895295 31973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:57.896461 31973 hybrid_clock.cc:648] HybridClock initialized: now 1790462637896457 us; error 35 us; skew 500 ppm
Sep 26 22:43:57 dist-test-slave-j8sq krb5kdc[31960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:57.898830 31973 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:43:57.899299 31973 webserver.cc:460] Webserver started at http://127.0.0.1:35625/ using document root <none> and password file <none>
I20260926 22:43:57.899487 31973 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:57.899524 31973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:57.899607 31973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:57.900556 31973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "6dfe06460b3e4df08000cb6e28b4bd8b"
format_stamp: "Formatted at 2026-09-26 22:43:57 on dist-test-slave-j8sq"
server_key: "8c9e47fbfd9672eedd49684f97c87b75"
server_key_iv: "b2bad4cb0c6d167a6af91f4a9efbdff0"
server_key_version: "encryptionkey@0"
I20260926 22:43:57.900913 31973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "6dfe06460b3e4df08000cb6e28b4bd8b"
format_stamp: "Formatted at 2026-09-26 22:43:57 on dist-test-slave-j8sq"
server_key: "8c9e47fbfd9672eedd49684f97c87b75"
server_key_iv: "b2bad4cb0c6d167a6af91f4a9efbdff0"
server_key_version: "encryptionkey@0"
I20260926 22:43:57.902287 31973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:43:57.903007 31988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:57.903234 31973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:57.903314 31973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "6dfe06460b3e4df08000cb6e28b4bd8b"
format_stamp: "Formatted at 2026-09-26 22:43:57 on dist-test-slave-j8sq"
server_key: "8c9e47fbfd9672eedd49684f97c87b75"
server_key_iv: "b2bad4cb0c6d167a6af91f4a9efbdff0"
server_key_version: "encryptionkey@0"
I20260926 22:43:57.903366 31973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:57.925737 31973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:57.927682 31973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:57.927789 31973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:57.931483 31973 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38167
I20260926 22:43:57.931530 32040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38167 every 8 connection(s)
I20260926 22:43:57.931850 31973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:43:57.932695 32041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:57.934963 32041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b: Bootstrap starting.
I20260926 22:43:57.935518 32041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:57.935803 32041 log.cc:824] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:57.936481 32041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b: No bootstrap required, opened a new log
I20260926 22:43:57.938050 32041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfe06460b3e4df08000cb6e28b4bd8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38167 } }
I20260926 22:43:57.938215 32041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:57.938265 32041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dfe06460b3e4df08000cb6e28b4bd8b, State: Initialized, Role: FOLLOWER
I20260926 22:43:57.938401 32041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfe06460b3e4df08000cb6e28b4bd8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38167 } }
I20260926 22:43:57.938480 32041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:43:57.938522 32041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:43:57.938570 32041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:57.939056 32041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfe06460b3e4df08000cb6e28b4bd8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38167 } }
I20260926 22:43:57.939184 32041 leader_election.cc:302] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dfe06460b3e4df08000cb6e28b4bd8b; no voters:
I20260926 22:43:57.939355 32041 leader_election.cc:288] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:43:57.939393 32044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:57.939594 32041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:43:57.939615 32044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [term 1 LEADER]: Becoming Leader. State: Replica: 6dfe06460b3e4df08000cb6e28b4bd8b, State: Running, Role: LEADER
I20260926 22:43:57.939692 32044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfe06460b3e4df08000cb6e28b4bd8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38167 } }
I20260926 22:43:57.940171 32045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dfe06460b3e4df08000cb6e28b4bd8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfe06460b3e4df08000cb6e28b4bd8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38167 } } }
I20260926 22:43:57.940254 32045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:57.940415 32046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dfe06460b3e4df08000cb6e28b4bd8b. Latest consensus state: current_term: 1 leader_uuid: "6dfe06460b3e4df08000cb6e28b4bd8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfe06460b3e4df08000cb6e28b4bd8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38167 } } }
I20260926 22:43:57.940488 32046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:57.940443 32053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:43:57.941057 32053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:43:57.941339 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 31973
I20260926 22:43:57.941457 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:43:57.941648 11695 external_mini_cluster.cc:1428] Setting key a6b46dd1d7bc58c4f7634265bde2515f
I20260926 22:43:57.942662 32053 catalog_manager.cc:1347] Generated new cluster ID: c58cef14209646dab27cc5a403912fd7
I20260926 22:43:57.942718 32053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:43:57.948210 32064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:57.942419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40291 (local address 127.0.0.1:38167)
0926 22:43:57.942908 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:57.942916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:57.942945 (+ 29us) server_negotiation.cc:407] Connection header received
0926 22:43:57.943053 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:57.943060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:57.943109 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:57.943234 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:57.943495 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:57.944855 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:57.946015 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:57.946652 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:57.946763 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:57.947479 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:57.947483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:57.947484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:57.947711 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:57.947755 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:57.947775 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:57.947884 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:57.947977 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 22:43:57.948009 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":66,"threads_started":1}
I20260926 22:43:57.949139 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:43:57.953896 32064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:57.950508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38071 (local address 127.0.0.1:38167)
0926 22:43:57.950543 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:57.950544 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:57.950667 (+ 123us) server_negotiation.cc:407] Connection header received
0926 22:43:57.950706 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:57.950706 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:57.950719 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:57.950764 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:57.950864 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:57.951579 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:57.952633 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:43:57.952926 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:43:57.952983 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:43:57.953629 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:57.953630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:57.953630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:57.953635 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:57.953648 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:57.953652 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:57.953707 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:57.953799 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 22:43:57.953810 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:43:57.979682 32053 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:43:57.980438 32053 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:43:57.996636 32053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dfe06460b3e4df08000cb6e28b4bd8b: Generated new TSK 0
I20260926 22:43:57.996830 32053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:43:58.003758 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 31973
W20260926 22:43:58.007817 31958 connection.cc:535] client connection to 127.0.0.1:38167 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38167): Connection reset by peer (error 104)
2026-09-26T22:43:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:43:58 dist-test-slave-j8sq krb5kdc[32072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:43:58 dist-test-slave-j8sq krb5kdc[32072](info): set up 2 sockets
Sep 26 22:43:58 dist-test-slave-j8sq krb5kdc[32072](info): commencing operation
krb5kdc: starting...
W20260926 22:43:59.038990 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:43:59 dist-test-slave-j8sq krb5kdc[32072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:43:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:43:59.152069 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:43:59.231702 32085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:43:59.231899 32085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:43:59.231921 32085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:43:59.233064 32085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:43:59.233114 32085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:43:59.233129 32085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:43:59.233142 32085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:43:59.233155 32085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:43:59.234760 32085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:43:59.235008 32085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:43:59.235244 32085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:43:59.240372 32093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:59.240346 32091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:43:59.240350 32090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:43:59.240513 32085 server_base.cc:1034] running on GCE node
I20260926 22:43:59.240832 32085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:43:59.241132 32085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:43:59.242291 32085 hybrid_clock.cc:648] HybridClock initialized: now 1790462639242278 us; error 28 us; skew 500 ppm
Sep 26 22:43:59 dist-test-slave-j8sq krb5kdc[32072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:43:59.244756 32085 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:43:59.245249 32085 webserver.cc:460] Webserver started at http://127.0.0.1:36663/ using document root <none> and password file <none>
I20260926 22:43:59.245481 32085 fs_manager.cc:359] Metadata directory not provided
I20260926 22:43:59.245551 32085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:43:59.245755 32085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:43:59.246914 32085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "147dbc77c6a348a0a3696aa106120ac9"
format_stamp: "Formatted at 2026-09-26 22:43:59 on dist-test-slave-j8sq"
server_key: "16e74a43a00adc09f35d2bf97523ec35"
server_key_iv: "e590c4a1b5738716d207352de67ad068"
server_key_version: "encryptionkey@0"
I20260926 22:43:59.247234 32085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "147dbc77c6a348a0a3696aa106120ac9"
format_stamp: "Formatted at 2026-09-26 22:43:59 on dist-test-slave-j8sq"
server_key: "16e74a43a00adc09f35d2bf97523ec35"
server_key_iv: "e590c4a1b5738716d207352de67ad068"
server_key_version: "encryptionkey@0"
I20260926 22:43:59.248658 32085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:43:59.249466 32100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:43:59.249650 32085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:43:59.249744 32085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "147dbc77c6a348a0a3696aa106120ac9"
format_stamp: "Formatted at 2026-09-26 22:43:59 on dist-test-slave-j8sq"
server_key: "16e74a43a00adc09f35d2bf97523ec35"
server_key_iv: "e590c4a1b5738716d207352de67ad068"
server_key_version: "encryptionkey@0"
I20260926 22:43:59.249800 32085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:43:59.253091 32085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:43:59.255008 32085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:43:59.255115 32085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:43:59.259075 32085 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39261
I20260926 22:43:59.259102 32152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39261 every 8 connection(s)
I20260926 22:43:59.259413 32085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:43:59.260329 32153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:43:59.262472 32153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9: Bootstrap starting.
I20260926 22:43:59.262966 32153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9: Neither blocks nor log segments found. Creating new log.
I20260926 22:43:59.263191 32153 log.cc:824] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9: Log is configured to *not* fsync() on all Append() calls
I20260926 22:43:59.263834 32153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9: No bootstrap required, opened a new log
I20260926 22:43:59.265327 32153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147dbc77c6a348a0a3696aa106120ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39261 } }
I20260926 22:43:59.265453 32153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:43:59.265475 32153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147dbc77c6a348a0a3696aa106120ac9, State: Initialized, Role: FOLLOWER
I20260926 22:43:59.265570 32153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147dbc77c6a348a0a3696aa106120ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39261 } }
I20260926 22:43:59.265648 32153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:43:59.265684 32153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:43:59.265774 32153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:43:59.266094 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 32085
I20260926 22:43:59.266208 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:43:59.266394 11695 external_mini_cluster.cc:1428] Setting key 3ccd60698a20f623d97701d35f09c61f
I20260926 22:43:59.266412 32153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147dbc77c6a348a0a3696aa106120ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39261 } }
I20260926 22:43:59.266532 32153 leader_election.cc:302] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 147dbc77c6a348a0a3696aa106120ac9; no voters:
I20260926 22:43:59.266700 32153 leader_election.cc:288] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:43:59.266778 32156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:43:59.266959 32156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [term 1 LEADER]: Becoming Leader. State: Replica: 147dbc77c6a348a0a3696aa106120ac9, State: Running, Role: LEADER
I20260926 22:43:59.266961 32153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:43:59.267112 32156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147dbc77c6a348a0a3696aa106120ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39261 } }
I20260926 22:43:59.267414 32156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "147dbc77c6a348a0a3696aa106120ac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147dbc77c6a348a0a3696aa106120ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39261 } } }
I20260926 22:43:59.267490 32156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:59.267614 32156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 147dbc77c6a348a0a3696aa106120ac9. Latest consensus state: current_term: 1 leader_uuid: "147dbc77c6a348a0a3696aa106120ac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147dbc77c6a348a0a3696aa106120ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39261 } } }
I20260926 22:43:59.267673 32156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:43:59.267750 32162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:43:59.268319 32162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:43:59.270145 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:59.266832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36163 (local address 127.0.0.1:39261)
0926 22:43:59.267007 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:59.267017 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:59.267918 (+ 901us) server_negotiation.cc:407] Connection header received
0926 22:43:59.268388 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:59.268396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:59.268427 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:59.268548 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:59.269599 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:59.269603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:59.269603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:59.269608 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:59.269652 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:59.269667 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:59.269736 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:59.269914 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 22:43:59.269956 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260926 22:43:59.270318 32162 catalog_manager.cc:1347] Generated new cluster ID: 33352eee655941a4a69708bd6bc8ee7a
I20260926 22:43:59.270378 32162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:43:59.275027 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:43:59.278980 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:43:59.276512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34047 (local address 127.0.0.1:39261)
0926 22:43:59.276545 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:43:59.276546 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:43:59.278007 (+ 1461us) server_negotiation.cc:407] Connection header received
0926 22:43:59.278106 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:43:59.278107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:43:59.278112 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:43:59.278162 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:43:59.278754 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:43:59.278755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:43:59.278755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:43:59.278758 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:43:59.278772 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:43:59.278777 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:43:59.278828 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:43:59.278898 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 22:43:59.278910 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:43:59.297554 32162 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:43:59.297753 32162 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:43:59.310691 32162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 147dbc77c6a348a0a3696aa106120ac9: Generated new TSK 0
I20260926 22:43:59.310887 32162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:43:59.315749 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 32085
2026-09-26T22:43:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:43:59 dist-test-slave-j8sq krb5kdc[32184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:43:59 dist-test-slave-j8sq krb5kdc[32184](info): set up 2 sockets
Sep 26 22:43:59 dist-test-slave-j8sq krb5kdc[32184](info): commencing operation
krb5kdc: starting...
W20260926 22:44:00.346835 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:00 dist-test-slave-j8sq krb5kdc[32184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:44:00.462559 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44099
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:00.540748 32197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:00.540943 32197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:00.540966 32197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:00.542112 32197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:00.542153 32197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:00.542167 32197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:00.542179 32197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:00.542218 32197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:00.543663 32197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:00.543888 32197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:00.544142 32197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:00.548913 32202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:00.548964 32203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:00.548931 32205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:00.549176 32197 server_base.cc:1034] running on GCE node
I20260926 22:44:00.549429 32197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:00.549746 32197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:00.550936 32197 hybrid_clock.cc:648] HybridClock initialized: now 1790462640550912 us; error 41 us; skew 500 ppm
Sep 26 22:44:00 dist-test-slave-j8sq krb5kdc[32184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:00.553593 32197 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:44:00.554117 32197 webserver.cc:460] Webserver started at http://127.0.0.1:35697/ using document root <none> and password file <none>
I20260926 22:44:00.554364 32197 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:00.554442 32197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:00.554600 32197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:00.555902 32197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "8271e95b3c7b42eabee4f9d7a96ada14"
format_stamp: "Formatted at 2026-09-26 22:44:00 on dist-test-slave-j8sq"
server_key: "d1956198cd01852b69f94e0ca5f7f837"
server_key_iv: "7bfa8578b252436112b8e9cc1e5c2961"
server_key_version: "encryptionkey@0"
I20260926 22:44:00.556373 32197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "8271e95b3c7b42eabee4f9d7a96ada14"
format_stamp: "Formatted at 2026-09-26 22:44:00 on dist-test-slave-j8sq"
server_key: "d1956198cd01852b69f94e0ca5f7f837"
server_key_iv: "7bfa8578b252436112b8e9cc1e5c2961"
server_key_version: "encryptionkey@0"
I20260926 22:44:00.558275 32197 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 22:44:00.559233 32212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:00.559432 32197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:44:00.559536 32197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "8271e95b3c7b42eabee4f9d7a96ada14"
format_stamp: "Formatted at 2026-09-26 22:44:00 on dist-test-slave-j8sq"
server_key: "d1956198cd01852b69f94e0ca5f7f837"
server_key_iv: "7bfa8578b252436112b8e9cc1e5c2961"
server_key_version: "encryptionkey@0"
I20260926 22:44:00.559614 32197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:00.570201 32197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:00.572024 32197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:00.572211 32197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:00.576448 32197 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44099
I20260926 22:44:00.576519 32264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44099 every 8 connection(s)
I20260926 22:44:00.576840 32197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:00.577700 32265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:00.579927 32265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14: Bootstrap starting.
I20260926 22:44:00.580490 32265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:00.580772 32265 log.cc:824] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:00.581496 32265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14: No bootstrap required, opened a new log
I20260926 22:44:00.583058 32265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8271e95b3c7b42eabee4f9d7a96ada14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44099 } }
I20260926 22:44:00.583221 32265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:00.583267 32265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8271e95b3c7b42eabee4f9d7a96ada14, State: Initialized, Role: FOLLOWER
I20260926 22:44:00.583416 32265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8271e95b3c7b42eabee4f9d7a96ada14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44099 } }
I20260926 22:44:00.583498 32265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:00.583536 32265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:00.583586 32265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:00.584084 32265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8271e95b3c7b42eabee4f9d7a96ada14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44099 } }
I20260926 22:44:00.584200 32265 leader_election.cc:302] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8271e95b3c7b42eabee4f9d7a96ada14; no voters:
I20260926 22:44:00.584379 32265 leader_election.cc:288] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:00.584414 32268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:00.584621 32265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:00.584637 32268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [term 1 LEADER]: Becoming Leader. State: Replica: 8271e95b3c7b42eabee4f9d7a96ada14, State: Running, Role: LEADER
I20260926 22:44:00.584743 32268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8271e95b3c7b42eabee4f9d7a96ada14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44099 } }
I20260926 22:44:00.585322 32269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8271e95b3c7b42eabee4f9d7a96ada14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8271e95b3c7b42eabee4f9d7a96ada14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44099 } } }
I20260926 22:44:00.585409 32269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:00.585625 32277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:00.585566 32270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8271e95b3c7b42eabee4f9d7a96ada14. Latest consensus state: current_term: 1 leader_uuid: "8271e95b3c7b42eabee4f9d7a96ada14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8271e95b3c7b42eabee4f9d7a96ada14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44099 } } }
I20260926 22:44:00.586143 32270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:00.586352 32277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:00.586843 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 32197
I20260926 22:44:00.586957 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:00.587131 11695 external_mini_cluster.cc:1428] Setting key fbbf4bb2e72baf0143d364268fddd21d
I20260926 22:44:00.587976 32277 catalog_manager.cc:1347] Generated new cluster ID: a2e41c5c09ca48e59c15e0fc98b92a18
I20260926 22:44:00.588027 32277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:44:00 dist-test-slave-j8sq krb5kdc[32184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 22:44:00.598578 32277 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:00.599179 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:00.587578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55987 (local address 127.0.0.1:44099)
0926 22:44:00.590232 (+ 2654us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:00.590240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:00.590262 (+ 22us) server_negotiation.cc:407] Connection header received
0926 22:44:00.590347 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:00.590352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:00.590396 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:00.590525 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:00.591175 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:00.592094 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:00.592874 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:00.593274 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:00.593391 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:00.595677 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:00.595681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:00.595682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:00.595699 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:00.597353 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:00.597729 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:00.597732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:00.597732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:00.597819 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:00.598069 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:00.598070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:00.598070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:00.598364 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:00.598536 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:00.598804 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 22:44:00.598951 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2618,"thread_start_us":87,"threads_started":1}
I20260926 22:44:00.599287 32277 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:00.600051 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:00.608898 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:00.601504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32933 (local address 127.0.0.1:44099)
0926 22:44:00.601538 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:00.601540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:00.601649 (+ 109us) server_negotiation.cc:407] Connection header received
0926 22:44:00.601739 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:00.601740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:00.601749 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:00.601776 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:00.602479 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:00.603242 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:00.604360 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:00.604687 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:00.604725 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:00.606194 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:00.606196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:00.606196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:00.606203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:00.607691 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:00.608066 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:00.608067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:00.608068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:00.608114 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:00.608336 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:00.608337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:00.608337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:00.608451 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:00.608521 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:00.608722 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 22:44:00.608798 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:44:00.611961 32277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8271e95b3c7b42eabee4f9d7a96ada14: Generated new TSK 0
I20260926 22:44:00.612154 32277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:00.616437 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 32197
2026-09-26T22:44:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:00 dist-test-slave-j8sq krb5kdc[32296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:00 dist-test-slave-j8sq krb5kdc[32296](info): set up 2 sockets
Sep 26 22:44:00 dist-test-slave-j8sq krb5kdc[32296](info): commencing operation
krb5kdc: starting...
W20260926 22:44:01.643235 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:01 dist-test-slave-j8sq krb5kdc[32296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:44:01.764715 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:01.843730 32309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:01.843997 32309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:01.844038 32309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:01.845227 32309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:01.845316 32309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:01.845342 32309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:01.845372 32309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:01.845402 32309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:01.846920 32309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:01.847198 32309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:01.847429 32309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:01.852217 32314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:01.852214 32317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:01.852227 32315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:01.852533 32309 server_base.cc:1034] running on GCE node
I20260926 22:44:01.852803 32309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:01.853046 32309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:01.854231 32309 hybrid_clock.cc:648] HybridClock initialized: now 1790462641854205 us; error 35 us; skew 500 ppm
Sep 26 22:44:01 dist-test-slave-j8sq krb5kdc[32296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:01.856783 32309 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:44:01.857159 32309 webserver.cc:460] Webserver started at http://127.0.0.1:43703/ using document root <none> and password file <none>
I20260926 22:44:01.857329 32309 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:01.857368 32309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:01.857455 32309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:01.858512 32309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "e854a4786b3e495ba0869f50d750478a"
format_stamp: "Formatted at 2026-09-26 22:44:01 on dist-test-slave-j8sq"
server_key: "fd7484c86c001b204c4eb87186cbcc0b"
server_key_iv: "09ba8e5b343df2eab965416dfec2596e"
server_key_version: "encryptionkey@0"
I20260926 22:44:01.858896 32309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "e854a4786b3e495ba0869f50d750478a"
format_stamp: "Formatted at 2026-09-26 22:44:01 on dist-test-slave-j8sq"
server_key: "fd7484c86c001b204c4eb87186cbcc0b"
server_key_iv: "09ba8e5b343df2eab965416dfec2596e"
server_key_version: "encryptionkey@0"
I20260926 22:44:01.860147 32309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:44:01.860822 32324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:01.861016 32309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:44:01.861100 32309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "e854a4786b3e495ba0869f50d750478a"
format_stamp: "Formatted at 2026-09-26 22:44:01 on dist-test-slave-j8sq"
server_key: "fd7484c86c001b204c4eb87186cbcc0b"
server_key_iv: "09ba8e5b343df2eab965416dfec2596e"
server_key_version: "encryptionkey@0"
I20260926 22:44:01.861153 32309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:01.873330 32309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:01.875640 32309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:01.875752 32309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:01.879522 32309 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38499
I20260926 22:44:01.879576 32376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38499 every 8 connection(s)
I20260926 22:44:01.879894 32309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:01.880831 32377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:01.883213 32377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a: Bootstrap starting.
I20260926 22:44:01.883810 32377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:01.884114 32377 log.cc:824] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:01.884860 32377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a: No bootstrap required, opened a new log
I20260926 22:44:01.886514 32377 raft_consensus.cc:348] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e854a4786b3e495ba0869f50d750478a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38499 } }
I20260926 22:44:01.886677 32377 raft_consensus.cc:374] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:01.886723 32377 raft_consensus.cc:729] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e854a4786b3e495ba0869f50d750478a, State: Initialized, Role: FOLLOWER
I20260926 22:44:01.886886 32377 consensus_queue.cc:260] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e854a4786b3e495ba0869f50d750478a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38499 } }
I20260926 22:44:01.886965 32377 raft_consensus.cc:388] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:01.887009 32377 raft_consensus.cc:482] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:01.887059 32377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:01.887611 32377 raft_consensus.cc:504] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e854a4786b3e495ba0869f50d750478a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38499 } }
I20260926 22:44:01.887725 32377 leader_election.cc:302] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e854a4786b3e495ba0869f50d750478a; no voters:
I20260926 22:44:01.887915 32377 leader_election.cc:288] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:01.887961 32380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:01.888175 32380 raft_consensus.cc:686] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [term 1 LEADER]: Becoming Leader. State: Replica: e854a4786b3e495ba0869f50d750478a, State: Running, Role: LEADER
I20260926 22:44:01.888182 32377 sys_catalog.cc:564] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:01.888273 32380 consensus_queue.cc:237] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e854a4786b3e495ba0869f50d750478a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38499 } }
I20260926 22:44:01.888622 32381 sys_catalog.cc:455] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e854a4786b3e495ba0869f50d750478a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e854a4786b3e495ba0869f50d750478a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38499 } } }
I20260926 22:44:01.888664 32382 sys_catalog.cc:455] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e854a4786b3e495ba0869f50d750478a. Latest consensus state: current_term: 1 leader_uuid: "e854a4786b3e495ba0869f50d750478a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e854a4786b3e495ba0869f50d750478a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38499 } } }
I20260926 22:44:01.888743 32381 sys_catalog.cc:458] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:01.888818 32382 sys_catalog.cc:458] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:01.889055 32384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:01.889334 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 32309
I20260926 22:44:01.889469 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:01.889587 32384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:01.889674 11695 external_mini_cluster.cc:1428] Setting key d75eaee2462a310a6664925bace1e621
I20260926 22:44:01.891525 32384 catalog_manager.cc:1347] Generated new cluster ID: 9538c2f7e1c24f1e901de2b8e12f76eb
I20260926 22:44:01.891582 32384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:01.896569 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:01.890317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54985 (local address 127.0.0.1:38499)
0926 22:44:01.890462 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:01.890471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:01.891079 (+ 608us) server_negotiation.cc:407] Connection header received
0926 22:44:01.891244 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:01.891252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:01.891310 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:01.891441 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:01.891783 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:01.893200 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:01.894289 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:01.894885 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:01.894969 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:01.895751 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:01.895755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:01.895755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:01.895985 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:01.896023 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:01.896038 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:01.896089 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:01.896308 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 22:44:01.896365 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
I20260926 22:44:01.897382 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:01.914002 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:01.910179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43959 (local address 127.0.0.1:38499)
0926 22:44:01.910247 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:01.910249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:01.910394 (+ 145us) server_negotiation.cc:407] Connection header received
0926 22:44:01.910467 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:01.910468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:01.910490 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:01.910550 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:01.910710 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:01.911461 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:01.912536 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:01.912864 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:01.912925 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:01.913685 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:01.913686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:01.913687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:01.913692 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:01.913708 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:01.913722 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:01.913776 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:01.913864 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 22:44:01.913894 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":40}
I20260926 22:44:01.914700 32384 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:01.915225 32384 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:01.919123 32384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e854a4786b3e495ba0869f50d750478a: Generated new TSK 0
I20260926 22:44:01.919304 32384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:01.925410 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 32309
2026-09-26T22:44:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:01 dist-test-slave-j8sq krb5kdc[32408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:01 dist-test-slave-j8sq krb5kdc[32408](info): set up 2 sockets
Sep 26 22:44:01 dist-test-slave-j8sq krb5kdc[32408](info): commencing operation
krb5kdc: starting...
W20260926 22:44:02.962939 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:03 dist-test-slave-j8sq krb5kdc[32408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:44:03.081290 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:03.160550 32421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:03.160797 32421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:03.160836 32421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:03.161991 32421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:03.162056 32421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:03.162078 32421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:03.162101 32421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:03.162125 32421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:03.163599 32421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:03.163863 32421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:03.164083 32421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:03.168746 32427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:03.168732 32429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:03.168953 32426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:03.168944 32421 server_base.cc:1034] running on GCE node
I20260926 22:44:03.169209 32421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:03.169457 32421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:03.170655 32421 hybrid_clock.cc:648] HybridClock initialized: now 1790462643170629 us; error 41 us; skew 500 ppm
Sep 26 22:44:03 dist-test-slave-j8sq krb5kdc[32408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:03.176494 32421 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:44:03.176905 32421 webserver.cc:460] Webserver started at http://127.0.0.1:33475/ using document root <none> and password file <none>
I20260926 22:44:03.177119 32421 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:03.177220 32421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:03.177342 32421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:03.178449 32421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "13cd758fbebb4d35843ff1f8e257e2de"
format_stamp: "Formatted at 2026-09-26 22:44:03 on dist-test-slave-j8sq"
server_key: "fe4703fd28cb1242ede241ab971a1c80"
server_key_iv: "25e5469a6ee5ecd54116d29ffe233cd4"
server_key_version: "encryptionkey@0"
I20260926 22:44:03.178758 32421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "13cd758fbebb4d35843ff1f8e257e2de"
format_stamp: "Formatted at 2026-09-26 22:44:03 on dist-test-slave-j8sq"
server_key: "fe4703fd28cb1242ede241ab971a1c80"
server_key_iv: "25e5469a6ee5ecd54116d29ffe233cd4"
server_key_version: "encryptionkey@0"
I20260926 22:44:03.180034 32421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:44:03.180775 32436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:03.180971 32421 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:44:03.181046 32421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "13cd758fbebb4d35843ff1f8e257e2de"
format_stamp: "Formatted at 2026-09-26 22:44:03 on dist-test-slave-j8sq"
server_key: "fe4703fd28cb1242ede241ab971a1c80"
server_key_iv: "25e5469a6ee5ecd54116d29ffe233cd4"
server_key_version: "encryptionkey@0"
I20260926 22:44:03.181154 32421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:03.184549 32421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:03.186954 32421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:03.187115 32421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:03.191016 32421 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45895
I20260926 22:44:03.191046 32488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45895 every 8 connection(s)
I20260926 22:44:03.191397 32421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:03.192415 32489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:03.194819 32489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de: Bootstrap starting.
I20260926 22:44:03.195384 32489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:03.195665 32489 log.cc:824] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:03.195819 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 32421
I20260926 22:44:03.195955 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:03.196139 11695 external_mini_cluster.cc:1428] Setting key d46d29d702e13868c7c86b81bd3036aa
I20260926 22:44:03.196408 32489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de: No bootstrap required, opened a new log
I20260926 22:44:03.198196 32489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cd758fbebb4d35843ff1f8e257e2de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260926 22:44:03.198350 32489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:03.198411 32489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13cd758fbebb4d35843ff1f8e257e2de, State: Initialized, Role: FOLLOWER
I20260926 22:44:03.198565 32489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cd758fbebb4d35843ff1f8e257e2de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260926 22:44:03.198661 32489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:03.198707 32489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:03.198753 32489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:03.199450 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:03.196686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49045 (local address 127.0.0.1:45895)
0926 22:44:03.196884 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:03.196893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:03.196924 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:44:03.198188 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:03.198195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:03.198225 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:03.198375 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:03.199000 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:03.199003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:03.199004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:03.199007 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:03.199049 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:03.199070 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:03.199149 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:03.199234 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 22:44:03.199274 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260926 22:44:03.199752 32489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cd758fbebb4d35843ff1f8e257e2de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260926 22:44:03.199864 32489 leader_election.cc:302] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13cd758fbebb4d35843ff1f8e257e2de; no voters:
I20260926 22:44:03.200014 32489 leader_election.cc:288] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:03.200066 32494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:03.200224 32494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [term 1 LEADER]: Becoming Leader. State: Replica: 13cd758fbebb4d35843ff1f8e257e2de, State: Running, Role: LEADER
I20260926 22:44:03.200270 32489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:03.200299 32494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cd758fbebb4d35843ff1f8e257e2de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260926 22:44:03.200318 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:03.200568 32495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13cd758fbebb4d35843ff1f8e257e2de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cd758fbebb4d35843ff1f8e257e2de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } } }
I20260926 22:44:03.200644 32495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:03.200789 32496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13cd758fbebb4d35843ff1f8e257e2de. Latest consensus state: current_term: 1 leader_uuid: "13cd758fbebb4d35843ff1f8e257e2de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cd758fbebb4d35843ff1f8e257e2de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } } }
I20260926 22:44:03.200836 32496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:03.200937 32499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:03.201404 32499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:03.203619 32499 catalog_manager.cc:1347] Generated new cluster ID: d0551a3f0257463fb6bee4e3ae9b4031
I20260926 22:44:03.203671 32499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:03.204028 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:03.202805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39069 (local address 127.0.0.1:45895)
0926 22:44:03.202843 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:03.202845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:03.203049 (+ 204us) server_negotiation.cc:407] Connection header received
0926 22:44:03.203118 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:03.203119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:03.203122 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:03.203199 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:03.203808 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:03.203809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:03.203810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:03.203814 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:03.203828 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:03.203833 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:03.203883 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:03.203936 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 22:44:03.203949 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:44:03.226853 32499 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:03.227052 32499 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:03.232951 32499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13cd758fbebb4d35843ff1f8e257e2de: Generated new TSK 0
I20260926 22:44:03.233146 32499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:03.237816 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 32421
2026-09-26T22:44:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:03 dist-test-slave-j8sq krb5kdc[32520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:03 dist-test-slave-j8sq krb5kdc[32520](info): set up 2 sockets
Sep 26 22:44:03 dist-test-slave-j8sq krb5kdc[32520](info): commencing operation
krb5kdc: starting...
W20260926 22:44:04.286873 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:04 dist-test-slave-j8sq krb5kdc[32520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:44:04.400091 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:04.478485 32533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:04.478708 32533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:04.478745 32533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:04.479900 32533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:04.479964 32533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:04.479987 32533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:04.480019 32533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:04.480043 32533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:04.481458 32533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:04.481724 32533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:04.481981 32533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:04.486506 32538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:04.486533 32541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:04.486519 32539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:04.486660 32533 server_base.cc:1034] running on GCE node
I20260926 22:44:04.486884 32533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:04.487195 32533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:04.488358 32533 hybrid_clock.cc:648] HybridClock initialized: now 1790462644488360 us; error 44 us; skew 500 ppm
Sep 26 22:44:04 dist-test-slave-j8sq krb5kdc[32520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:04.490887 32533 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:44:04.491307 32533 webserver.cc:460] Webserver started at http://127.0.0.1:41985/ using document root <none> and password file <none>
I20260926 22:44:04.491526 32533 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:04.491569 32533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:04.491732 32533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:04.492722 32533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "f73f071273294159b90b6d7454daf8df"
format_stamp: "Formatted at 2026-09-26 22:44:04 on dist-test-slave-j8sq"
server_key: "8103cf0e2125c4a759de37ac28e82d46"
server_key_iv: "6ebcd26b7bd8ffba63dca20b4819aa11"
server_key_version: "encryptionkey@0"
I20260926 22:44:04.493053 32533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "f73f071273294159b90b6d7454daf8df"
format_stamp: "Formatted at 2026-09-26 22:44:04 on dist-test-slave-j8sq"
server_key: "8103cf0e2125c4a759de37ac28e82d46"
server_key_iv: "6ebcd26b7bd8ffba63dca20b4819aa11"
server_key_version: "encryptionkey@0"
I20260926 22:44:04.494410 32533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:44:04.495112 32548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:04.495275 32533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:04.495373 32533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "f73f071273294159b90b6d7454daf8df"
format_stamp: "Formatted at 2026-09-26 22:44:04 on dist-test-slave-j8sq"
server_key: "8103cf0e2125c4a759de37ac28e82d46"
server_key_iv: "6ebcd26b7bd8ffba63dca20b4819aa11"
server_key_version: "encryptionkey@0"
I20260926 22:44:04.495447 32533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:04.507974 32533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:04.510135 32533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:04.510285 32533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:04.514037 32533 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33209
I20260926 22:44:04.514076 32600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33209 every 8 connection(s)
I20260926 22:44:04.514382 32533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:04.515127 32601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:04.517278 32601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df: Bootstrap starting.
I20260926 22:44:04.517865 32601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:04.518154 32601 log.cc:824] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:04.518812 32601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df: No bootstrap required, opened a new log
I20260926 22:44:04.520356 32601 raft_consensus.cc:348] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73f071273294159b90b6d7454daf8df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33209 } }
I20260926 22:44:04.520509 32601 raft_consensus.cc:374] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:04.520552 32601 raft_consensus.cc:729] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73f071273294159b90b6d7454daf8df, State: Initialized, Role: FOLLOWER
I20260926 22:44:04.520715 32601 consensus_queue.cc:260] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73f071273294159b90b6d7454daf8df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33209 } }
I20260926 22:44:04.520802 32601 raft_consensus.cc:388] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:04.520838 32601 raft_consensus.cc:482] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:04.520887 32601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:04.521406 32601 raft_consensus.cc:504] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73f071273294159b90b6d7454daf8df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33209 } }
I20260926 22:44:04.521519 32601 leader_election.cc:302] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f73f071273294159b90b6d7454daf8df; no voters:
I20260926 22:44:04.521700 32601 leader_election.cc:288] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:04.521745 32604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:04.521971 32601 sys_catalog.cc:564] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:04.521971 32604 raft_consensus.cc:686] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [term 1 LEADER]: Becoming Leader. State: Replica: f73f071273294159b90b6d7454daf8df, State: Running, Role: LEADER
I20260926 22:44:04.522181 32604 consensus_queue.cc:237] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73f071273294159b90b6d7454daf8df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33209 } }
I20260926 22:44:04.522856 32606 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [sys.catalog]: SysCatalogTable state changed. Reason: New leader f73f071273294159b90b6d7454daf8df. Latest consensus state: current_term: 1 leader_uuid: "f73f071273294159b90b6d7454daf8df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73f071273294159b90b6d7454daf8df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33209 } } }
I20260926 22:44:04.522936 32606 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:04.522987 32605 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f73f071273294159b90b6d7454daf8df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73f071273294159b90b6d7454daf8df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33209 } } }
I20260926 22:44:04.523067 32605 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:04.523384 32614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:04.523870 32614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:04.524227 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 32533
I20260926 22:44:04.524344 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:04.524528 11695 external_mini_cluster.cc:1428] Setting key ab29e5240b0fee8d73f41d8602c2076c
I20260926 22:44:04.525341 32614 catalog_manager.cc:1347] Generated new cluster ID: a6d04b51561441789a6143bc34674896
I20260926 22:44:04.525411 32614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:44:04 dist-test-slave-j8sq krb5kdc[32520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 22:44:04.532560 32614 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:04.533156 32614 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:04.536682 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:04.524976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:41183 (local address 127.0.0.1:33209)
0926 22:44:04.525615 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:04.525623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:04.525656 (+ 33us) server_negotiation.cc:407] Connection header received
0926 22:44:04.525761 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:04.525768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:04.525868 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:04.526004 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:04.526755 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:04.528234 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:04.529319 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:04.529974 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:04.530086 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:04.532955 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:04.532958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:04.532959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:04.532971 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:04.535074 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:04.535487 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:04.535490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:04.535490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:04.535594 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:04.535825 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:04.535826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:04.535826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:04.536091 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:04.536247 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:04.536356 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 22:44:04.536454 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":65,"threads_started":1}
I20260926 22:44:04.537639 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:04.540688 32614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f73f071273294159b90b6d7454daf8df: Generated new TSK 0
I20260926 22:44:04.540884 32614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:04.545637 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:04.539103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:55787 (local address 127.0.0.1:33209)
0926 22:44:04.539144 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:04.539146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:04.539232 (+ 86us) server_negotiation.cc:407] Connection header received
0926 22:44:04.539256 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:04.539257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:04.539266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:04.539332 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:04.539876 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:04.540672 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:04.541441 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:04.541759 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:04.541869 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:04.543038 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:04.543039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:04.543040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:04.543046 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:04.544431 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:04.544775 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:04.544776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:04.544777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:04.544855 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:04.545140 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:04.545141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:04.545141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:04.545265 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:04.545368 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:04.545489 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 22:44:04.545550 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":23}
I20260926 22:44:04.547668 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 32533
W20260926 22:44:04.552205 32518 connection.cc:535] client connection to 127.0.0.1:33209 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33209): Connection reset by peer (error 104)
2026-09-26T22:44:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:04 dist-test-slave-j8sq krb5kdc[32632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:04 dist-test-slave-j8sq krb5kdc[32632](info): set up 2 sockets
Sep 26 22:44:04 dist-test-slave-j8sq krb5kdc[32632](info): commencing operation
krb5kdc: starting...
W20260926 22:44:05.575274 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:05 dist-test-slave-j8sq krb5kdc[32632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:44:05.692767 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37107
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:05.772778 32645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:05.773021 32645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:05.773061 32645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:05.774246 32645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:05.774323 32645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:05.774350 32645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:05.774384 32645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:05.774418 32645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:05.775882 32645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:05.776151 32645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:05.776391 32645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:05.781298 32650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:05.781327 32651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:05.781303 32653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:05.781385 32645 server_base.cc:1034] running on GCE node
I20260926 22:44:05.781875 32645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:05.782188 32645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:05.783349 32645 hybrid_clock.cc:648] HybridClock initialized: now 1790462645783329 us; error 37 us; skew 500 ppm
Sep 26 22:44:05 dist-test-slave-j8sq krb5kdc[32632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:05.785943 32645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:44:05.786391 32645 webserver.cc:460] Webserver started at http://127.0.0.1:46307/ using document root <none> and password file <none>
I20260926 22:44:05.786605 32645 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:05.786677 32645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:05.786823 32645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:05.787855 32645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "0019cdbc734541418b8bcdf132162289"
format_stamp: "Formatted at 2026-09-26 22:44:05 on dist-test-slave-j8sq"
server_key: "16e01b95ca709b53f9f8dd5d8ff45388"
server_key_iv: "2e51325c2bf4be4c61d040757b6e507c"
server_key_version: "encryptionkey@0"
I20260926 22:44:05.788206 32645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "0019cdbc734541418b8bcdf132162289"
format_stamp: "Formatted at 2026-09-26 22:44:05 on dist-test-slave-j8sq"
server_key: "16e01b95ca709b53f9f8dd5d8ff45388"
server_key_iv: "2e51325c2bf4be4c61d040757b6e507c"
server_key_version: "encryptionkey@0"
I20260926 22:44:05.789590 32645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:44:05.790524 32660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:05.790755 32645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:44:05.790851 32645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "0019cdbc734541418b8bcdf132162289"
format_stamp: "Formatted at 2026-09-26 22:44:05 on dist-test-slave-j8sq"
server_key: "16e01b95ca709b53f9f8dd5d8ff45388"
server_key_iv: "2e51325c2bf4be4c61d040757b6e507c"
server_key_version: "encryptionkey@0"
I20260926 22:44:05.790934 32645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:05.800179 32645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:05.802033 32645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:05.802183 32645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:05.806161 32645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37107
I20260926 22:44:05.806192 32712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37107 every 8 connection(s)
I20260926 22:44:05.806524 32645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:05.807137 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 32645
I20260926 22:44:05.807260 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:05.807336 32713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:05.807469 11695 external_mini_cluster.cc:1428] Setting key 3cca31bfe05ab179d3d2f777a5de79a2
I20260926 22:44:05.810518 32713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289: Bootstrap starting.
I20260926 22:44:05.811249 32713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:05.811549 32713 log.cc:824] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:05.812217 32713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289: No bootstrap required, opened a new log
I20260926 22:44:05.813153 32716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:05.807984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:38259 (local address 127.0.0.1:37107)
0926 22:44:05.808161 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:05.808172 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:05.808199 (+ 27us) server_negotiation.cc:407] Connection header received
0926 22:44:05.808621 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:05.808626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:05.808674 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:05.808782 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:05.809046 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:05.809974 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:05.811089 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:05.811535 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:05.811612 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:05.812481 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:05.812485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:05.812485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:05.812674 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:05.812712 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:05.812727 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:05.812812 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:05.812903 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 22:44:05.812927 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260926 22:44:05.814339 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:05.814721 32713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0019cdbc734541418b8bcdf132162289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37107 } }
I20260926 22:44:05.814847 32713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:05.814868 32713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0019cdbc734541418b8bcdf132162289, State: Initialized, Role: FOLLOWER
I20260926 22:44:05.815068 32713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0019cdbc734541418b8bcdf132162289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37107 } }
I20260926 22:44:05.815172 32718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:44:05.815267 32718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0019cdbc734541418b8bcdf132162289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37107 } }
I20260926 22:44:05.815414 32718 leader_election.cc:302] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0019cdbc734541418b8bcdf132162289; no voters:
I20260926 22:44:05.815418 32713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:05.815508 32713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:05.815550 32713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:05.816291 32713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0019cdbc734541418b8bcdf132162289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37107 } }
I20260926 22:44:05.816368 32713 leader_election.cc:302] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0019cdbc734541418b8bcdf132162289; no voters:
I20260926 22:44:05.816491 32713 leader_election.cc:288] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:05.816547 32724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:05.816723 32724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [term 1 LEADER]: Becoming Leader. State: Replica: 0019cdbc734541418b8bcdf132162289, State: Running, Role: LEADER
I20260926 22:44:05.816730 32713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:05.816857 32718 leader_election.cc:288] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260926 22:44:05.817952 32724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0019cdbc734541418b8bcdf132162289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37107 } }
I20260926 22:44:05.818409 32718 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [term 1 LEADER]: Leader pre-election decision vote started in defunct term 0: won
I20260926 22:44:05.818571 32726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0019cdbc734541418b8bcdf132162289. Latest consensus state: current_term: 1 leader_uuid: "0019cdbc734541418b8bcdf132162289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0019cdbc734541418b8bcdf132162289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37107 } } }
I20260926 22:44:05.818667 32726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:05.818638 32725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0019cdbc734541418b8bcdf132162289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0019cdbc734541418b8bcdf132162289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37107 } } }
I20260926 22:44:05.818715 32725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:05.819016 32740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:05.819550 32716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:05.815759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:35625 (local address 127.0.0.1:37107)
0926 22:44:05.815819 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:05.815821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:05.815890 (+ 69us) server_negotiation.cc:407] Connection header received
0926 22:44:05.815959 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:05.815960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:05.815970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:05.816027 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:05.816123 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:05.816823 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:05.817620 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:05.817959 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:05.818023 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:05.819249 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:05.819250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:05.819250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:05.819254 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:05.819268 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:05.819272 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:05.819336 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:05.819447 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 22:44:05.819457 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 22:44:05.819574 32740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:05.821002 32740 catalog_manager.cc:1347] Generated new cluster ID: 36283ced6bc644329b8ef31886d87eee
I20260926 22:44:05.821043 32740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:05.829829 32740 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:05.830387 32740 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:05.834211 32740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0019cdbc734541418b8bcdf132162289: Generated new TSK 0
I20260926 22:44:05.834388 32740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:05.845057 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 32645
W20260926 22:44:05.849376 32630 connection.cc:535] client connection to 127.0.0.1:37107 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37107): Connection reset by peer (error 104)
2026-09-26T22:44:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:05 dist-test-slave-j8sq krb5kdc[32746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:05 dist-test-slave-j8sq krb5kdc[32746](info): set up 2 sockets
Sep 26 22:44:05 dist-test-slave-j8sq krb5kdc[32746](info): commencing operation
krb5kdc: starting...
W20260926 22:44:06.871093 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:06 dist-test-slave-j8sq krb5kdc[32746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:44:06.982349 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:07.060837 32759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:07.061089 32759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:07.061131 32759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:07.062860 32759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:07.062955 32759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:07.062979 32759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:07.063009 32759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:07.063053 32759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:07.064584 32759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:07.064862 32759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:07.065096 32759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:07.069802 32765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:07.069815 32764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:07.069823 32767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:07.070048 32759 server_base.cc:1034] running on GCE node
I20260926 22:44:07.070312 32759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:07.070596 32759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:07.071777 32759 hybrid_clock.cc:648] HybridClock initialized: now 1790462647071761 us; error 24 us; skew 500 ppm
Sep 26 22:44:07 dist-test-slave-j8sq krb5kdc[32746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:07.074312 32759 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:44:07.074745 32759 webserver.cc:460] Webserver started at http://127.0.0.1:33485/ using document root <none> and password file <none>
I20260926 22:44:07.074962 32759 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:07.075028 32759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:07.075174 32759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:07.076236 32759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "9d86d4ca268e4e6faeca8f836c70002f"
format_stamp: "Formatted at 2026-09-26 22:44:07 on dist-test-slave-j8sq"
server_key: "1908fc847d3aa98dfee7f4acb18e6886"
server_key_iv: "ada2b0e096bb2fb490b252e9f96407cc"
server_key_version: "encryptionkey@0"
I20260926 22:44:07.076587 32759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "9d86d4ca268e4e6faeca8f836c70002f"
format_stamp: "Formatted at 2026-09-26 22:44:07 on dist-test-slave-j8sq"
server_key: "1908fc847d3aa98dfee7f4acb18e6886"
server_key_iv: "ada2b0e096bb2fb490b252e9f96407cc"
server_key_version: "encryptionkey@0"
I20260926 22:44:07.077977 32759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:44:07.078819 306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:07.078970 32759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:44:07.079098 32759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "9d86d4ca268e4e6faeca8f836c70002f"
format_stamp: "Formatted at 2026-09-26 22:44:07 on dist-test-slave-j8sq"
server_key: "1908fc847d3aa98dfee7f4acb18e6886"
server_key_iv: "ada2b0e096bb2fb490b252e9f96407cc"
server_key_version: "encryptionkey@0"
I20260926 22:44:07.079178 32759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:07.082520 32759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:07.084862 32759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:07.085022 32759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:07.089212 32759 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44371
I20260926 22:44:07.089269 358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44371 every 8 connection(s)
I20260926 22:44:07.089613 32759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:07.090680 359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:07.093479 359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f: Bootstrap starting.
I20260926 22:44:07.094138 359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:07.094437 359 log.cc:824] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:07.095125 359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f: No bootstrap required, opened a new log
I20260926 22:44:07.096314 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 32759
I20260926 22:44:07.096439 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:07.096612 11695 external_mini_cluster.cc:1428] Setting key 3322d6ae571083a7d4cdde869ba442ac
I20260926 22:44:07.097055 359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d86d4ca268e4e6faeca8f836c70002f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44371 } }
I20260926 22:44:07.097263 359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:07.097345 359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d86d4ca268e4e6faeca8f836c70002f, State: Initialized, Role: FOLLOWER
I20260926 22:44:07.097546 359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d86d4ca268e4e6faeca8f836c70002f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44371 } }
I20260926 22:44:07.097646 359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:07.097680 359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:07.097755 359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:07.098598 359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d86d4ca268e4e6faeca8f836c70002f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44371 } }
I20260926 22:44:07.098713 359 leader_election.cc:302] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d86d4ca268e4e6faeca8f836c70002f; no voters:
I20260926 22:44:07.098910 359 leader_election.cc:288] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:07.098941 364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:07.098994 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:07.097251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:41551 (local address 127.0.0.1:44371)
0926 22:44:07.097448 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:07.097457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:07.097502 (+ 45us) server_negotiation.cc:407] Connection header received
0926 22:44:07.097611 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:07.097618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:07.097648 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:07.097807 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:07.098494 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:07.098498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:07.098499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:07.098503 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:07.098547 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:07.098562 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:07.098651 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:07.098738 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 22:44:07.098781 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260926 22:44:07.099129 364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [term 1 LEADER]: Becoming Leader. State: Replica: 9d86d4ca268e4e6faeca8f836c70002f, State: Running, Role: LEADER
I20260926 22:44:07.099161 359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:07.099934 364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d86d4ca268e4e6faeca8f836c70002f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44371 } }
I20260926 22:44:07.100214 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:07.100615 366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d86d4ca268e4e6faeca8f836c70002f. Latest consensus state: current_term: 1 leader_uuid: "9d86d4ca268e4e6faeca8f836c70002f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d86d4ca268e4e6faeca8f836c70002f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44371 } } }
I20260926 22:44:07.100766 365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d86d4ca268e4e6faeca8f836c70002f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d86d4ca268e4e6faeca8f836c70002f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44371 } } }
I20260926 22:44:07.100884 366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:07.100891 365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:07.101217 379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:07.101716 379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:07.103237 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:07.101916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:43841 (local address 127.0.0.1:44371)
0926 22:44:07.101953 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:07.101955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:07.102051 (+ 96us) server_negotiation.cc:407] Connection header received
0926 22:44:07.102125 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:07.102126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:07.102130 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:07.102192 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:07.102960 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:07.102961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:07.102962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:07.102975 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:07.102990 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:07.102994 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:07.103064 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:07.103133 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 22:44:07.103147 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:44:07.103551 379 catalog_manager.cc:1347] Generated new cluster ID: 1f770ea3eea44651ba3567f35a7d05ba
I20260926 22:44:07.103653 379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:07.122812 379 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:07.122961 379 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:07.142316 379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d86d4ca268e4e6faeca8f836c70002f: Generated new TSK 0
I20260926 22:44:07.142520 379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:07.146209 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 32759
2026-09-26T22:44:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:07 dist-test-slave-j8sq krb5kdc[390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:07 dist-test-slave-j8sq krb5kdc[390](info): set up 2 sockets
Sep 26 22:44:07 dist-test-slave-j8sq krb5kdc[390](info): commencing operation
krb5kdc: starting...
W20260926 22:44:08.186962 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:08 dist-test-slave-j8sq krb5kdc[390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:44:08.301824 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:08.380519 403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:08.380748 403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:08.380784 403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:08.381933 403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:08.381995 403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:08.382017 403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:08.382036 403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:08.382066 403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:08.383518 403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:08.383775 403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:08.383980 403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:08.388592 408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:08.388626 403 server_base.cc:1034] running on GCE node
W20260926 22:44:08.388695 409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:08.388651 411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:08.389130 403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:08.389396 403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:08.390549 403 hybrid_clock.cc:648] HybridClock initialized: now 1790462648390538 us; error 32 us; skew 500 ppm
Sep 26 22:44:08 dist-test-slave-j8sq krb5kdc[390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:08.397555 403 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:44:08.398034 403 webserver.cc:460] Webserver started at http://127.0.0.1:43821/ using document root <none> and password file <none>
I20260926 22:44:08.398218 403 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:08.398257 403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:08.398343 403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:08.399335 403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "58bab5bf7c664f8d9fbbdeff82fdcbec"
format_stamp: "Formatted at 2026-09-26 22:44:08 on dist-test-slave-j8sq"
server_key: "a1ebdd9ef7d84f25a7ef678804daacb0"
server_key_iv: "f5b2e45507f57836fcfeacff3c86559e"
server_key_version: "encryptionkey@0"
I20260926 22:44:08.399619 403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "58bab5bf7c664f8d9fbbdeff82fdcbec"
format_stamp: "Formatted at 2026-09-26 22:44:08 on dist-test-slave-j8sq"
server_key: "a1ebdd9ef7d84f25a7ef678804daacb0"
server_key_iv: "f5b2e45507f57836fcfeacff3c86559e"
server_key_version: "encryptionkey@0"
I20260926 22:44:08.400848 403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:44:08.401510 418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:08.401717 403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:44:08.401801 403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "58bab5bf7c664f8d9fbbdeff82fdcbec"
format_stamp: "Formatted at 2026-09-26 22:44:08 on dist-test-slave-j8sq"
server_key: "a1ebdd9ef7d84f25a7ef678804daacb0"
server_key_iv: "f5b2e45507f57836fcfeacff3c86559e"
server_key_version: "encryptionkey@0"
I20260926 22:44:08.401925 403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:08.416792 403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:08.418622 403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:08.418764 403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:08.422557 403 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45371
I20260926 22:44:08.422621 470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45371 every 8 connection(s)
I20260926 22:44:08.422930 403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:08.423717 471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:08.425987 471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec: Bootstrap starting.
I20260926 22:44:08.425993 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 403
I20260926 22:44:08.426091 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:08.426260 11695 external_mini_cluster.cc:1428] Setting key 8bc1f7b4ddf2650f8dc54da22ef0869a
I20260926 22:44:08.426666 471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:08.426993 471 log.cc:824] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:08.427702 471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec: No bootstrap required, opened a new log
I20260926 22:44:08.429801 471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bab5bf7c664f8d9fbbdeff82fdcbec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } }
I20260926 22:44:08.429996 471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:08.430020 471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58bab5bf7c664f8d9fbbdeff82fdcbec, State: Initialized, Role: FOLLOWER
I20260926 22:44:08.430159 471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bab5bf7c664f8d9fbbdeff82fdcbec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } }
I20260926 22:44:08.430223 471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:08.430243 471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:08.430305 471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:08.430871 471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bab5bf7c664f8d9fbbdeff82fdcbec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } }
I20260926 22:44:08.430964 471 leader_election.cc:302] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58bab5bf7c664f8d9fbbdeff82fdcbec; no voters:
I20260926 22:44:08.431107 471 leader_election.cc:288] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:08.431159 476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:08.431346 471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:08.431365 476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [term 1 LEADER]: Becoming Leader. State: Replica: 58bab5bf7c664f8d9fbbdeff82fdcbec, State: Running, Role: LEADER
I20260926 22:44:08.431450 476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bab5bf7c664f8d9fbbdeff82fdcbec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } }
I20260926 22:44:08.431727 476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58bab5bf7c664f8d9fbbdeff82fdcbec. Latest consensus state: current_term: 1 leader_uuid: "58bab5bf7c664f8d9fbbdeff82fdcbec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bab5bf7c664f8d9fbbdeff82fdcbec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } } }
I20260926 22:44:08.431821 476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:08.431737 477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58bab5bf7c664f8d9fbbdeff82fdcbec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58bab5bf7c664f8d9fbbdeff82fdcbec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } } }
I20260926 22:44:08.431948 477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:08.432808 480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:08.433308 480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 22:44:08 dist-test-slave-j8sq krb5kdc[390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 22:44:08.434813 480 catalog_manager.cc:1347] Generated new cluster ID: 25b466e2097a4a329b2bf939e998d6f1
I20260926 22:44:08.434862 480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:08.438231 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:08.426770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:57387 (local address 127.0.0.1:45371)
0926 22:44:08.426958 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:08.426968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:08.426999 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:44:08.427539 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:08.427547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:08.427625 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:08.427765 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:08.428368 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:08.429719 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:08.430781 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:08.431395 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:08.431840 (+ 445us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:08.434512 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:08.434515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:08.434516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:08.434540 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:08.436545 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:08.436987 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:08.436990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:08.436990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:08.437094 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:08.437317 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:08.437318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:08.437318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:08.437624 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:08.437797 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:08.437924 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 22:44:08.438022 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260926 22:44:08.439077 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:08.447485 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:08.440544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:32781 (local address 127.0.0.1:45371)
0926 22:44:08.440596 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:08.440598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:08.440608 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:44:08.440630 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:08.440631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:08.440641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:08.440713 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:08.441266 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:08.442049 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:08.443010 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:08.443337 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:08.443404 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:08.444690 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:08.444692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:08.444692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:08.444698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:08.446232 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:08.446622 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:08.446623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:08.446623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:08.446678 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:08.446883 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:08.446883 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:08.446884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:08.447004 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:08.447095 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:08.447238 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 22:44:08.447300 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:44:08.458118 480 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:08.458606 480 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:08.462069 480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58bab5bf7c664f8d9fbbdeff82fdcbec: Generated new TSK 0
I20260926 22:44:08.462261 480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:08.469552 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 403
W20260926 22:44:08.473258 388 connection.cc:535] client connection to 127.0.0.1:45371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45371): Connection reset by peer (error 104)
2026-09-26T22:44:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:08 dist-test-slave-j8sq krb5kdc[502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:08 dist-test-slave-j8sq krb5kdc[502](info): set up 2 sockets
Sep 26 22:44:08 dist-test-slave-j8sq krb5kdc[502](info): commencing operation
krb5kdc: starting...
W20260926 22:44:09.535228 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:09 dist-test-slave-j8sq krb5kdc[502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:44:09.650453 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:09.732622 515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:09.732811 515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:09.732833 515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:09.734005 515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:09.734046 515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:09.734061 515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:09.734074 515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:09.734086 515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:09.735544 515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:09.735776 515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:09.735983 515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:09.740607 520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:09.740638 521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:09.740687 515 server_base.cc:1034] running on GCE node
W20260926 22:44:09.740614 523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:09.741144 515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:09.741417 515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:09.742571 515 hybrid_clock.cc:648] HybridClock initialized: now 1790462649742553 us; error 38 us; skew 500 ppm
Sep 26 22:44:09 dist-test-slave-j8sq krb5kdc[502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:09.744972 515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:44:09.745345 515 webserver.cc:460] Webserver started at http://127.0.0.1:43749/ using document root <none> and password file <none>
I20260926 22:44:09.745522 515 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:09.745560 515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:09.745656 515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:09.746698 515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "759c6f840aca44698740a6db9d78e65d"
format_stamp: "Formatted at 2026-09-26 22:44:09 on dist-test-slave-j8sq"
server_key: "58450db7087d8992c6dd39349a37b0d4"
server_key_iv: "3c5a6f4a2168a622488b6bc16da959aa"
server_key_version: "encryptionkey@0"
I20260926 22:44:09.747010 515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "759c6f840aca44698740a6db9d78e65d"
format_stamp: "Formatted at 2026-09-26 22:44:09 on dist-test-slave-j8sq"
server_key: "58450db7087d8992c6dd39349a37b0d4"
server_key_iv: "3c5a6f4a2168a622488b6bc16da959aa"
server_key_version: "encryptionkey@0"
I20260926 22:44:09.748288 515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:44:09.748946 530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:09.749109 515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:44:09.749193 515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "759c6f840aca44698740a6db9d78e65d"
format_stamp: "Formatted at 2026-09-26 22:44:09 on dist-test-slave-j8sq"
server_key: "58450db7087d8992c6dd39349a37b0d4"
server_key_iv: "3c5a6f4a2168a622488b6bc16da959aa"
server_key_version: "encryptionkey@0"
I20260926 22:44:09.749246 515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:09.760007 515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:09.762181 515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:09.762290 515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:09.765797 515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45825
I20260926 22:44:09.765901 582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45825 every 8 connection(s)
I20260926 22:44:09.766248 515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:09.767119 583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:09.769239 583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d: Bootstrap starting.
I20260926 22:44:09.769829 583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:09.770128 583 log.cc:824] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:09.770817 583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d: No bootstrap required, opened a new log
I20260926 22:44:09.772356 583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759c6f840aca44698740a6db9d78e65d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45825 } }
I20260926 22:44:09.772565 583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:09.772631 583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759c6f840aca44698740a6db9d78e65d, State: Initialized, Role: FOLLOWER
I20260926 22:44:09.772766 583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759c6f840aca44698740a6db9d78e65d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45825 } }
I20260926 22:44:09.772845 583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:09.772868 583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:09.772903 583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:09.773350 583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759c6f840aca44698740a6db9d78e65d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45825 } }
I20260926 22:44:09.773427 583 leader_election.cc:302] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 759c6f840aca44698740a6db9d78e65d; no voters:
I20260926 22:44:09.773538 583 leader_election.cc:288] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:09.773576 586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:09.773756 586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [term 1 LEADER]: Becoming Leader. State: Replica: 759c6f840aca44698740a6db9d78e65d, State: Running, Role: LEADER
I20260926 22:44:09.773818 583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:09.773895 586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759c6f840aca44698740a6db9d78e65d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45825 } }
I20260926 22:44:09.774260 588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 759c6f840aca44698740a6db9d78e65d. Latest consensus state: current_term: 1 leader_uuid: "759c6f840aca44698740a6db9d78e65d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759c6f840aca44698740a6db9d78e65d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45825 } } }
I20260926 22:44:09.774380 587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "759c6f840aca44698740a6db9d78e65d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759c6f840aca44698740a6db9d78e65d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45825 } } }
I20260926 22:44:09.774399 588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:09.774464 587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:09.774773 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 515
I20260926 22:44:09.774839 595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:09.774878 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:09.775058 11695 external_mini_cluster.cc:1428] Setting key 726f279d2257a3b8ecf7131eb01d9afe
I20260926 22:44:09.775302 595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:09.776777 595 catalog_manager.cc:1347] Generated new cluster ID: a0ed9fee431b4002aaac3d1bd93f1e26
I20260926 22:44:09.776823 595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:09.781365 605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:09.775670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:49733 (local address 127.0.0.1:45825)
0926 22:44:09.776043 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:09.776052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:09.776245 (+ 193us) server_negotiation.cc:407] Connection header received
0926 22:44:09.776381 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:09.776388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:09.776461 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:09.776602 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:09.776842 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:09.778232 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:09.779285 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:09.779926 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:09.780030 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:09.780707 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:09.780711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:09.780711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:09.780912 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:09.780948 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:09.780962 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:09.781050 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:09.781127 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 22:44:09.781154 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":63,"threads_started":1}
I20260926 22:44:09.782281 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:09.785899 595 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:09.786535 595 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:09.787508 605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:09.783901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:55139 (local address 127.0.0.1:45825)
0926 22:44:09.783951 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:09.783953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:09.783961 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:44:09.784052 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:09.784053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:09.784062 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:09.784115 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:09.784221 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:09.785109 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:09.786152 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:09.786494 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:09.786561 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:09.787221 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:09.787222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:09.787223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:09.787226 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:09.787239 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:09.787243 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:09.787293 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:09.787429 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 22:44:09.787439 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":183,"server-negotiator.queue_time_us":33}
I20260926 22:44:09.791684 595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 759c6f840aca44698740a6db9d78e65d: Generated new TSK 0
I20260926 22:44:09.791839 595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:09.797964 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 515
W20260926 22:44:09.802029 500 connection.cc:535] client connection to 127.0.0.1:45825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45825): Connection reset by peer (error 104)
2026-09-26T22:44:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:09 dist-test-slave-j8sq krb5kdc[614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:09 dist-test-slave-j8sq krb5kdc[614](info): set up 2 sockets
Sep 26 22:44:09 dist-test-slave-j8sq krb5kdc[614](info): commencing operation
krb5kdc: starting...
W20260926 22:44:10.826982 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:10 dist-test-slave-j8sq krb5kdc[614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:44:10.936739 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:11.015736 627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:11.015928 627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:11.015952 627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:11.017120 627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:11.017163 627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:11.017179 627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:11.017191 627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:11.017203 627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:11.018678 627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35025
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:11.018913 627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:11.019143 627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:11.023980 632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:11.024029 627 server_base.cc:1034] running on GCE node
W20260926 22:44:11.023976 635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:11.024031 633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:11.024394 627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:11.024662 627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:11.025799 627 hybrid_clock.cc:648] HybridClock initialized: now 1790462651025788 us; error 31 us; skew 500 ppm
Sep 26 22:44:11 dist-test-slave-j8sq krb5kdc[614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:11.028316 627 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:44:11.028721 627 webserver.cc:460] Webserver started at http://127.0.0.1:41437/ using document root <none> and password file <none>
I20260926 22:44:11.028929 627 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:11.028991 627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:11.029134 627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:11.030242 627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "28970dc2b56041ec842cfb990962d6ac"
format_stamp: "Formatted at 2026-09-26 22:44:11 on dist-test-slave-j8sq"
server_key: "d21c33bba2a519893513983fafdae7dd"
server_key_iv: "a0a391b489bdd2f5773e37f7603f744a"
server_key_version: "encryptionkey@0"
I20260926 22:44:11.030581 627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "28970dc2b56041ec842cfb990962d6ac"
format_stamp: "Formatted at 2026-09-26 22:44:11 on dist-test-slave-j8sq"
server_key: "d21c33bba2a519893513983fafdae7dd"
server_key_iv: "a0a391b489bdd2f5773e37f7603f744a"
server_key_version: "encryptionkey@0"
I20260926 22:44:11.031896 627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:44:11.032574 642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:11.032737 627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:44:11.032827 627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "28970dc2b56041ec842cfb990962d6ac"
format_stamp: "Formatted at 2026-09-26 22:44:11 on dist-test-slave-j8sq"
server_key: "d21c33bba2a519893513983fafdae7dd"
server_key_iv: "a0a391b489bdd2f5773e37f7603f744a"
server_key_version: "encryptionkey@0"
I20260926 22:44:11.032902 627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:11.036065 627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:11.038218 627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:11.038362 627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:11.042179 627 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35025
I20260926 22:44:11.042228 694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35025 every 8 connection(s)
I20260926 22:44:11.042546 627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:11.043349 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:11.045469 695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac: Bootstrap starting.
I20260926 22:44:11.046047 695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:11.046284 695 log.cc:824] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:11.046911 695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac: No bootstrap required, opened a new log
I20260926 22:44:11.048420 695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28970dc2b56041ec842cfb990962d6ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35025 } }
I20260926 22:44:11.048550 695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:11.048573 695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28970dc2b56041ec842cfb990962d6ac, State: Initialized, Role: FOLLOWER
I20260926 22:44:11.048709 695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28970dc2b56041ec842cfb990962d6ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35025 } }
I20260926 22:44:11.048799 695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:11.048827 695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:11.048854 695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:11.049320 695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28970dc2b56041ec842cfb990962d6ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35025 } }
I20260926 22:44:11.049400 695 leader_election.cc:302] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28970dc2b56041ec842cfb990962d6ac; no voters:
I20260926 22:44:11.049510 695 leader_election.cc:288] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:11.049585 698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:11.049744 695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:11.049783 698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [term 1 LEADER]: Becoming Leader. State: Replica: 28970dc2b56041ec842cfb990962d6ac, State: Running, Role: LEADER
I20260926 22:44:11.049945 698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28970dc2b56041ec842cfb990962d6ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35025 } }
I20260926 22:44:11.050387 700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28970dc2b56041ec842cfb990962d6ac. Latest consensus state: current_term: 1 leader_uuid: "28970dc2b56041ec842cfb990962d6ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28970dc2b56041ec842cfb990962d6ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35025 } } }
I20260926 22:44:11.050487 700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:11.050519 699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28970dc2b56041ec842cfb990962d6ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28970dc2b56041ec842cfb990962d6ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35025 } } }
I20260926 22:44:11.050629 699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:11.050796 707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:11.051174 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 627
I20260926 22:44:11.051306 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:11.051319 707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:11.051543 11695 external_mini_cluster.cc:1428] Setting key f8361991888f33a31f39b21585f0cdf7
I20260926 22:44:11.053107 707 catalog_manager.cc:1347] Generated new cluster ID: f0dedd177fee478aaefdd6992f016ec6
I20260926 22:44:11.053172 707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:11.054139 717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:11.051986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:38441 (local address 127.0.0.1:35025)
0926 22:44:11.052160 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:11.052173 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:11.052350 (+ 177us) server_negotiation.cc:407] Connection header received
0926 22:44:11.052537 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:11.052545 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:11.052583 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:11.052748 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:11.053631 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:11.053635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:11.053635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:11.053640 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:11.053695 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:11.053710 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:11.053786 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:11.053889 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 22:44:11.053936 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260926 22:44:11.055039 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:11.071133 717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:11.058304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.219:42011 (local address 127.0.0.1:35025)
0926 22:44:11.058340 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:11.058342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:11.070051 (+ 11709us) server_negotiation.cc:407] Connection header received
0926 22:44:11.070122 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:11.070123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:11.070130 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:11.070206 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:11.070878 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:11.070879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:11.070880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:11.070885 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:11.070902 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:11.070907 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:11.070966 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:11.071031 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 22:44:11.071047 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:44:11.073524 707 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:11.073709 707 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:11.078982 707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28970dc2b56041ec842cfb990962d6ac: Generated new TSK 0
I20260926 22:44:11.079154 707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:11.081856 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 627
2026-09-26T22:44:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1284 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:11 dist-test-slave-j8sq krb5kdc[726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:11 dist-test-slave-j8sq krb5kdc[726](info): set up 2 sockets
Sep 26 22:44:11 dist-test-slave-j8sq krb5kdc[726](info): commencing operation
krb5kdc: starting...
W20260926 22:44:12.122887 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:12 dist-test-slave-j8sq krb5kdc[726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:44:12.238821 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:42777
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:42777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:12.317936 739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:12.318149 739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:12.318172 739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:12.319245 739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:12.319281 739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:12.319295 739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:12.319307 739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:12.319319 739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:12.320674 739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:42777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:42777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:12.320843 739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:12.321007 739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:12.325632 745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:12.325608 744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:12.325646 739 server_base.cc:1034] running on GCE node
W20260926 22:44:12.325613 747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:12.326092 739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:12.326409 739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:12.327625 739 hybrid_clock.cc:648] HybridClock initialized: now 1790462652327608 us; error 41 us; skew 500 ppm
Sep 26 22:44:12 dist-test-slave-j8sq krb5kdc[726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462652, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:12.330092 739 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:44:12.330458 739 webserver.cc:460] Webserver started at http://127.11.107.254:40599/ using document root <none> and password file <none>
I20260926 22:44:12.330652 739 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:12.330693 739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:12.330776 739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:12.331770 739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "70537f7a019d42a39cbe4626d4bf3a7b"
format_stamp: "Formatted at 2026-09-26 22:44:12 on dist-test-slave-j8sq"
server_key: "37343d731e01b2f1ca9daca0846b9c2d"
server_key_iv: "4b6c7675f24d33feebedb14ce2e52238"
server_key_version: "encryptionkey@0"
I20260926 22:44:12.332070 739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "70537f7a019d42a39cbe4626d4bf3a7b"
format_stamp: "Formatted at 2026-09-26 22:44:12 on dist-test-slave-j8sq"
server_key: "37343d731e01b2f1ca9daca0846b9c2d"
server_key_iv: "4b6c7675f24d33feebedb14ce2e52238"
server_key_version: "encryptionkey@0"
I20260926 22:44:12.333333 739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:44:12.334033 754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:12.334192 739 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:44:12.334275 739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "70537f7a019d42a39cbe4626d4bf3a7b"
format_stamp: "Formatted at 2026-09-26 22:44:12 on dist-test-slave-j8sq"
server_key: "37343d731e01b2f1ca9daca0846b9c2d"
server_key_iv: "4b6c7675f24d33feebedb14ce2e52238"
server_key_version: "encryptionkey@0"
I20260926 22:44:12.334326 739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:12.355544 739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:12.357434 739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:12.357542 739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:12.361235 739 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:42777
I20260926 22:44:12.361310 806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:42777 every 8 connection(s)
I20260926 22:44:12.361651 739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:12.362557 807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:12.363415 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 739
I20260926 22:44:12.363528 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:12.363750 11695 external_mini_cluster.cc:1428] Setting key 1d1e1759342b98dbe0b7868aae41b607
I20260926 22:44:12.365916 807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b: Bootstrap starting.
I20260926 22:44:12.366497 807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:12.366798 807 log.cc:824] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:12.367437 807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b: No bootstrap required, opened a new log
I20260926 22:44:12.369345 807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70537f7a019d42a39cbe4626d4bf3a7b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 42777 } }
I20260926 22:44:12.369489 807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:12.369511 807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70537f7a019d42a39cbe4626d4bf3a7b, State: Initialized, Role: FOLLOWER
I20260926 22:44:12.369658 807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70537f7a019d42a39cbe4626d4bf3a7b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 42777 } }
I20260926 22:44:12.369745 807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:12.369789 807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:12.369881 807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [term 0 FOLLOWER]: Advancing to term 1
Sep 26 22:44:12 dist-test-slave-j8sq krb5kdc[726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:44:12.370434 807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70537f7a019d42a39cbe4626d4bf3a7b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 42777 } }
I20260926 22:44:12.370548 807 leader_election.cc:302] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70537f7a019d42a39cbe4626d4bf3a7b; no voters:
I20260926 22:44:12.370710 807 leader_election.cc:288] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:12.370770 812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:12.370922 812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [term 1 LEADER]: Becoming Leader. State: Replica: 70537f7a019d42a39cbe4626d4bf3a7b, State: Running, Role: LEADER
I20260926 22:44:12.370970 807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:12.371040 812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70537f7a019d42a39cbe4626d4bf3a7b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 42777 } }
I20260926 22:44:12.371374 812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70537f7a019d42a39cbe4626d4bf3a7b. Latest consensus state: current_term: 1 leader_uuid: "70537f7a019d42a39cbe4626d4bf3a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70537f7a019d42a39cbe4626d4bf3a7b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 42777 } } }
I20260926 22:44:12.371582 812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:12.371512 813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70537f7a019d42a39cbe4626d4bf3a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70537f7a019d42a39cbe4626d4bf3a7b" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 42777 } } }
I20260926 22:44:12.371965 813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:12.372030 821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:12.372570 821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:12.374402 810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:12.364230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59521 (local address 127.11.107.254:42777)
0926 22:44:12.364392 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:12.364400 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:12.364513 (+ 113us) server_negotiation.cc:407] Connection header received
0926 22:44:12.364958 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:12.364965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:12.365009 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:12.365118 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:12.365912 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:12.366967 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:12.367770 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:12.368202 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:12.368314 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:12.370654 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:12.370658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:12.370658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:12.370671 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:12.372922 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:12.373295 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:12.373298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:12.373299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:12.373392 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:12.373610 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:12.373611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:12.373612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:12.373889 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:12.374050 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:12.374148 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 22:44:12.374248 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260926 22:44:12.375334 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:12.378299 821 catalog_manager.cc:1347] Generated new cluster ID: 21a346909dff4ca2b3365c800270af25
I20260926 22:44:12.378355 821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:12.383081 810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:12.376749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50247 (local address 127.11.107.254:42777)
0926 22:44:12.376803 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:12.376805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:12.376849 (+ 44us) server_negotiation.cc:407] Connection header received
0926 22:44:12.376921 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:12.376921 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:12.376930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:12.376983 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:12.377462 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:12.378460 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:12.379137 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:12.379377 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:12.379448 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:12.380555 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:12.380556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:12.380556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:12.380564 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:12.381904 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:12.382254 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:12.382255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:12.382256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:12.382305 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:12.382516 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:12.382517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:12.382517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:12.382618 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:12.382735 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:12.382862 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 22:44:12.382921 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:44:12.404438 821 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:12.405171 821 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:12.409312 821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70537f7a019d42a39cbe4626d4bf3a7b: Generated new TSK 0
I20260926 22:44:12.409538 821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:12.413666 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 739
W20260926 22:44:12.417999 724 connection.cc:535] client connection to 127.11.107.254:42777 recv error: Network error: failed to read from TLS socket (remote: 127.11.107.254:42777): Connection reset by peer (error 104)
2026-09-26T22:44:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:12 dist-test-slave-j8sq krb5kdc[838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:12 dist-test-slave-j8sq krb5kdc[838](info): set up 2 sockets
Sep 26 22:44:12 dist-test-slave-j8sq krb5kdc[838](info): commencing operation
krb5kdc: starting...
W20260926 22:44:13.443099 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:13 dist-test-slave-j8sq krb5kdc[838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:44:13.558773 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:41901
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:41901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:13.639571 851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:13.639775 851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:13.639797 851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:13.640913 851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:13.640952 851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:13.640966 851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:13.640980 851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:13.641023 851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:13.642498 851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:41901
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:41901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:13.642732 851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:13.642939 851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:13.647639 857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:13.647711 851 server_base.cc:1034] running on GCE node
W20260926 22:44:13.647686 856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:13.647688 859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:13.648041 851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:13.648357 851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:13.649507 851 hybrid_clock.cc:648] HybridClock initialized: now 1790462653649502 us; error 33 us; skew 500 ppm
Sep 26 22:44:13 dist-test-slave-j8sq krb5kdc[838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462653, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:13.656984 851 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:44:13.657415 851 webserver.cc:460] Webserver started at http://127.11.107.254:33309/ using document root <none> and password file <none>
I20260926 22:44:13.657630 851 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:13.657693 851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:13.657831 851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:13.658883 851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "7bb61b39d9c84d689ebbb72f0e4f80d5"
format_stamp: "Formatted at 2026-09-26 22:44:13 on dist-test-slave-j8sq"
server_key: "5053a33527b8012f99601c4e8ff698ee"
server_key_iv: "5bbcfcbe697438b780c1c186598d981a"
server_key_version: "encryptionkey@0"
I20260926 22:44:13.659220 851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "7bb61b39d9c84d689ebbb72f0e4f80d5"
format_stamp: "Formatted at 2026-09-26 22:44:13 on dist-test-slave-j8sq"
server_key: "5053a33527b8012f99601c4e8ff698ee"
server_key_iv: "5bbcfcbe697438b780c1c186598d981a"
server_key_version: "encryptionkey@0"
I20260926 22:44:13.660509 851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:44:13.661242 866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:13.661417 851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:44:13.661509 851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "7bb61b39d9c84d689ebbb72f0e4f80d5"
format_stamp: "Formatted at 2026-09-26 22:44:13 on dist-test-slave-j8sq"
server_key: "5053a33527b8012f99601c4e8ff698ee"
server_key_iv: "5bbcfcbe697438b780c1c186598d981a"
server_key_version: "encryptionkey@0"
I20260926 22:44:13.661579 851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:13.674261 851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:13.676393 851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:13.676541 851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:13.681515 851 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:41901
I20260926 22:44:13.681566 918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:41901 every 8 connection(s)
I20260926 22:44:13.682086 851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:13.683013 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 851
I20260926 22:44:13.683136 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:13.683125 919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:13.683332 11695 external_mini_cluster.cc:1428] Setting key 7a79891f0d922b05b34a3664a5dcb2c4
I20260926 22:44:13.686322 919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5: Bootstrap starting.
I20260926 22:44:13.687016 919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:13.687311 919 log.cc:824] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:13.687956 919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5: No bootstrap required, opened a new log
I20260926 22:44:13.689189 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:13.683800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48493 (local address 127.11.107.254:41901)
0926 22:44:13.683976 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:13.683986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:13.684021 (+ 35us) server_negotiation.cc:407] Connection header received
0926 22:44:13.684640 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:13.684647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:13.684697 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:13.684805 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:13.685028 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:13.685927 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:13.687056 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:13.687670 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:13.687789 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:13.688470 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:13.688475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:13.688475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:13.688644 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:13.688673 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:13.688684 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:13.688772 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:13.688853 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 22:44:13.688881 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260926 22:44:13.689507 919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb61b39d9c84d689ebbb72f0e4f80d5" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41901 } }
I20260926 22:44:13.689707 919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:13.689757 919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb61b39d9c84d689ebbb72f0e4f80d5, State: Initialized, Role: FOLLOWER
I20260926 22:44:13.689927 919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb61b39d9c84d689ebbb72f0e4f80d5" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41901 } }
I20260926 22:44:13.690028 919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:13.690083 919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:13.690133 919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:13.690284 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:13.690804 919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb61b39d9c84d689ebbb72f0e4f80d5" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41901 } }
I20260926 22:44:13.690922 919 leader_election.cc:302] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bb61b39d9c84d689ebbb72f0e4f80d5; no voters:
I20260926 22:44:13.691092 919 leader_election.cc:288] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:13.691161 924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:13.691337 924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [term 1 LEADER]: Becoming Leader. State: Replica: 7bb61b39d9c84d689ebbb72f0e4f80d5, State: Running, Role: LEADER
I20260926 22:44:13.691385 919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:13.691432 924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb61b39d9c84d689ebbb72f0e4f80d5" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41901 } }
I20260926 22:44:13.691879 925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bb61b39d9c84d689ebbb72f0e4f80d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb61b39d9c84d689ebbb72f0e4f80d5" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41901 } } }
I20260926 22:44:13.692032 925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:13.692269 927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bb61b39d9c84d689ebbb72f0e4f80d5. Latest consensus state: current_term: 1 leader_uuid: "7bb61b39d9c84d689ebbb72f0e4f80d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb61b39d9c84d689ebbb72f0e4f80d5" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 41901 } } }
I20260926 22:44:13.692334 938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:13.692412 927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:13.692898 938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:13.694519 938 catalog_manager.cc:1347] Generated new cluster ID: aff951722a7a40fa8fc1e69c1d98dad1
I20260926 22:44:13.694573 938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:13.698661 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:13.691802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58689 (local address 127.11.107.254:41901)
0926 22:44:13.691838 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:13.691840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:13.692950 (+ 1110us) server_negotiation.cc:407] Connection header received
0926 22:44:13.693004 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:13.693005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:13.693015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:13.693075 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:13.693264 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:13.693774 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:13.694709 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:13.694934 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:13.695007 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:13.698402 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:13.698403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:13.698403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:13.698410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:13.698425 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:13.698430 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:13.698504 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:13.698568 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 22:44:13.698577 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:44:13.728425 938 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:13.729009 938 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:13.739964 938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bb61b39d9c84d689ebbb72f0e4f80d5: Generated new TSK 0
I20260926 22:44:13.740170 938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:13.742614 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 851
W20260926 22:44:13.747541 836 connection.cc:535] client connection to 127.11.107.254:41901 recv error: Network error: failed to read from TLS socket (remote: 127.11.107.254:41901): Connection reset by peer (error 104)
2026-09-26T22:44:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:13 dist-test-slave-j8sq krb5kdc[950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:13 dist-test-slave-j8sq krb5kdc[950](info): set up 2 sockets
Sep 26 22:44:13 dist-test-slave-j8sq krb5kdc[950](info): commencing operation
krb5kdc: starting...
W20260926 22:44:14.766913 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:14 dist-test-slave-j8sq krb5kdc[950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:44:14.878690 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:33857
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:33857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:14.957825 963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:14.958026 963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:14.958048 963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:14.959213 963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:14.959252 963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:14.959266 963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:14.959280 963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:14.959338 963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:14.960726 963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:33857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:33857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:14.960970 963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:14.961174 963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:14.965860 969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:14.965919 968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:14.965901 963 server_base.cc:1034] running on GCE node
W20260926 22:44:14.965878 971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:14.966310 963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:14.966609 963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:14.967768 963 hybrid_clock.cc:648] HybridClock initialized: now 1790462654967757 us; error 34 us; skew 500 ppm
Sep 26 22:44:14 dist-test-slave-j8sq krb5kdc[950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462654, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:14.970175 963 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:44:14.970575 963 webserver.cc:460] Webserver started at http://127.11.107.254:41327/ using document root <none> and password file <none>
I20260926 22:44:14.970777 963 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:14.970836 963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:14.970968 963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:14.972235 963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "18d66bbbf8d043e5aca06db1b19c7978"
format_stamp: "Formatted at 2026-09-26 22:44:14 on dist-test-slave-j8sq"
server_key: "8398deffd760ce3c8535d58eef683780"
server_key_iv: "e238ffb75acba0521c952eaec34c051f"
server_key_version: "encryptionkey@0"
I20260926 22:44:14.972734 963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "18d66bbbf8d043e5aca06db1b19c7978"
format_stamp: "Formatted at 2026-09-26 22:44:14 on dist-test-slave-j8sq"
server_key: "8398deffd760ce3c8535d58eef683780"
server_key_iv: "e238ffb75acba0521c952eaec34c051f"
server_key_version: "encryptionkey@0"
I20260926 22:44:14.974174 963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:44:14.974884 978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:14.975054 963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:44:14.975160 963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "18d66bbbf8d043e5aca06db1b19c7978"
format_stamp: "Formatted at 2026-09-26 22:44:14 on dist-test-slave-j8sq"
server_key: "8398deffd760ce3c8535d58eef683780"
server_key_iv: "e238ffb75acba0521c952eaec34c051f"
server_key_version: "encryptionkey@0"
I20260926 22:44:14.975239 963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:14.978420 963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:14.980646 963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:14.980814 963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:14.984920 963 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:33857
I20260926 22:44:14.984959 1030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:33857 every 8 connection(s)
I20260926 22:44:14.985313 963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:14.986372 1031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:14.988430 1031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978: Bootstrap starting.
I20260926 22:44:14.988965 1031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:14.989231 1031 log.cc:824] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:14.989880 1031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978: No bootstrap required, opened a new log
I20260926 22:44:14.991407 1031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d66bbbf8d043e5aca06db1b19c7978" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 33857 } }
I20260926 22:44:14.991531 1031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:14.991552 1031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d66bbbf8d043e5aca06db1b19c7978, State: Initialized, Role: FOLLOWER
I20260926 22:44:14.991736 1031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d66bbbf8d043e5aca06db1b19c7978" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 33857 } }
I20260926 22:44:14.991811 1031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:14.991856 1031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:14.991907 1031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:14.992708 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 963
I20260926 22:44:14.992708 1031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d66bbbf8d043e5aca06db1b19c7978" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 33857 } }
I20260926 22:44:14.992817 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:14.992817 1031 leader_election.cc:302] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18d66bbbf8d043e5aca06db1b19c7978; no voters:
I20260926 22:44:14.992964 1031 leader_election.cc:288] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:14.992985 1034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:14.992995 11695 external_mini_cluster.cc:1428] Setting key a9b2f4d5fd4ae416af1fffa4c5421daa
I20260926 22:44:14.993090 1034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [term 1 LEADER]: Becoming Leader. State: Replica: 18d66bbbf8d043e5aca06db1b19c7978, State: Running, Role: LEADER
I20260926 22:44:14.993217 1031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:14.993230 1034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d66bbbf8d043e5aca06db1b19c7978" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 33857 } }
I20260926 22:44:14.993587 1034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18d66bbbf8d043e5aca06db1b19c7978. Latest consensus state: current_term: 1 leader_uuid: "18d66bbbf8d043e5aca06db1b19c7978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d66bbbf8d043e5aca06db1b19c7978" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 33857 } } }
I20260926 22:44:14.993599 1035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18d66bbbf8d043e5aca06db1b19c7978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d66bbbf8d043e5aca06db1b19c7978" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 33857 } } }
I20260926 22:44:14.993770 1034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:14.994061 1035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:14.994082 1045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:14.994706 1045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:14.996325 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:14.993605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39849 (local address 127.11.107.254:33857)
0926 22:44:14.993902 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:14.993911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:14.993947 (+ 36us) server_negotiation.cc:407] Connection header received
0926 22:44:14.995001 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:14.995008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:14.995037 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:14.995183 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:14.995791 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:14.995794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:14.995795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:14.995799 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:14.995864 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:14.995880 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:14.995960 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:14.996117 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 22:44:14.996158 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":88,"threads_started":1}
I20260926 22:44:14.996394 1045 catalog_manager.cc:1347] Generated new cluster ID: 8409b088e54249c79a0e59876be9e7fd
I20260926 22:44:14.996438 1045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:14.997154 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:15.003235 1045 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:15.003381 1045 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:15.010963 1045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18d66bbbf8d043e5aca06db1b19c7978: Generated new TSK 0
I20260926 22:44:15.011058 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:15.002121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33375 (local address 127.11.107.254:33857)
0926 22:44:15.002166 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:15.002168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:15.002382 (+ 214us) server_negotiation.cc:407] Connection header received
0926 22:44:15.002429 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:15.002430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:15.002435 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:15.002498 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:15.010691 (+ 8193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:15.010693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:15.010693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:15.010700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:15.010717 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:15.010722 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:15.010795 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:15.010911 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 22:44:15.010925 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 22:44:15.011209 1045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:15.018205 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 963
2026-09-26T22:44:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1274 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:15 dist-test-slave-j8sq krb5kdc[1062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:15 dist-test-slave-j8sq krb5kdc[1062](info): set up 2 sockets
Sep 26 22:44:15 dist-test-slave-j8sq krb5kdc[1062](info): commencing operation
krb5kdc: starting...
W20260926 22:44:16.055106 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:16 dist-test-slave-j8sq krb5kdc[1062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:44:16.168839 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:43893
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:43893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:16.247632 1075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:16.247864 1075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:16.247900 1075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:16.249024 1075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:16.249089 1075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:16.249112 1075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:16.249142 1075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:16.249164 1075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:16.250701 1075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:43893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:43893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:16.250963 1075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:16.251175 1075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:16.255911 1083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:16.256047 1075 server_base.cc:1034] running on GCE node
W20260926 22:44:16.255899 1081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:16.255877 1080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:16.256450 1075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:16.256716 1075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:16.257892 1075 hybrid_clock.cc:648] HybridClock initialized: now 1790462656257877 us; error 27 us; skew 500 ppm
Sep 26 22:44:16 dist-test-slave-j8sq krb5kdc[1062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462656, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:16.260495 1075 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:44:16.260972 1075 webserver.cc:460] Webserver started at http://127.11.107.254:35963/ using document root <none> and password file <none>
I20260926 22:44:16.261198 1075 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:16.261267 1075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:16.261420 1075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:16.262604 1075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "ad2066e881e74df1a338e084d7780056"
format_stamp: "Formatted at 2026-09-26 22:44:16 on dist-test-slave-j8sq"
server_key: "fe0d63ca73446217f10c732f9eb4d8a1"
server_key_iv: "15ca92cc539cc965205e29c4552d8016"
server_key_version: "encryptionkey@0"
I20260926 22:44:16.263042 1075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "ad2066e881e74df1a338e084d7780056"
format_stamp: "Formatted at 2026-09-26 22:44:16 on dist-test-slave-j8sq"
server_key: "fe0d63ca73446217f10c732f9eb4d8a1"
server_key_iv: "15ca92cc539cc965205e29c4552d8016"
server_key_version: "encryptionkey@0"
I20260926 22:44:16.264467 1075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:44:16.265308 1090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:16.265497 1075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:44:16.265632 1075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "ad2066e881e74df1a338e084d7780056"
format_stamp: "Formatted at 2026-09-26 22:44:16 on dist-test-slave-j8sq"
server_key: "fe0d63ca73446217f10c732f9eb4d8a1"
server_key_iv: "15ca92cc539cc965205e29c4552d8016"
server_key_version: "encryptionkey@0"
I20260926 22:44:16.265709 1075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:16.282356 1075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:16.284799 1075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:16.284968 1075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:16.289331 1075 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:43893
I20260926 22:44:16.289366 1142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:43893 every 8 connection(s)
I20260926 22:44:16.289754 1075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:16.290640 1143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:16.292878 1143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056: Bootstrap starting.
I20260926 22:44:16.293236 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 1075
I20260926 22:44:16.293351 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:16.293452 1143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:16.293582 11695 external_mini_cluster.cc:1428] Setting key d42749e0596e483ddb265905b49ef28b
I20260926 22:44:16.293802 1143 log.cc:824] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:16.294674 1143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056: No bootstrap required, opened a new log
I20260926 22:44:16.297080 1143 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2066e881e74df1a338e084d7780056" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43893 } }
I20260926 22:44:16.297259 1143 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:16.297298 1143 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2066e881e74df1a338e084d7780056, State: Initialized, Role: FOLLOWER
I20260926 22:44:16.297422 1143 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2066e881e74df1a338e084d7780056" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43893 } }
I20260926 22:44:16.297494 1143 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:16.297524 1143 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:16.297583 1143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:16.298285 1143 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2066e881e74df1a338e084d7780056" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43893 } }
I20260926 22:44:16.298405 1143 leader_election.cc:302] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad2066e881e74df1a338e084d7780056; no voters:
I20260926 22:44:16.298596 1143 leader_election.cc:288] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:16.298645 1148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:16.298821 1148 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [term 1 LEADER]: Becoming Leader. State: Replica: ad2066e881e74df1a338e084d7780056, State: Running, Role: LEADER
I20260926 22:44:16.298882 1143 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:16.298988 1148 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2066e881e74df1a338e084d7780056" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43893 } }
I20260926 22:44:16.299317 1149 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad2066e881e74df1a338e084d7780056. Latest consensus state: current_term: 1 leader_uuid: "ad2066e881e74df1a338e084d7780056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2066e881e74df1a338e084d7780056" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43893 } } }
I20260926 22:44:16.299443 1149 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:16.299875 1155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:16.300343 1155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 22:44:16 dist-test-slave-j8sq krb5kdc[1062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:44:16.302049 1155 catalog_manager.cc:1347] Generated new cluster ID: 2beabbbef5eb45398f1e468f58502301
I20260926 22:44:16.302516 1155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:16.301991 1150 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad2066e881e74df1a338e084d7780056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2066e881e74df1a338e084d7780056" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 43893 } } }
I20260926 22:44:16.306156 1150 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:16.308732 1147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:16.294129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33413 (local address 127.11.107.254:43893)
0926 22:44:16.294312 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:16.294324 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:16.294359 (+ 35us) server_negotiation.cc:407] Connection header received
0926 22:44:16.294792 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:16.294800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:16.294865 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:16.294981 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:16.295822 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:16.296676 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:16.297758 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:16.298215 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:16.298296 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:16.301409 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:16.301413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:16.301414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:16.301426 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:16.306701 (+ 5275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:16.307140 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:16.307143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:16.307143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:16.307240 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:16.307428 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:16.307429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:16.307429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:16.307707 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:16.307861 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:16.308372 (+ 511us) server_negotiation.cc:299] Negotiation successful
0926 22:44:16.308496 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260926 22:44:16.309667 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:16.314260 1155 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:16.314924 1155 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:16.318303 1147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:16.311196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38359 (local address 127.11.107.254:43893)
0926 22:44:16.311232 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:16.311234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:16.311260 (+ 26us) server_negotiation.cc:407] Connection header received
0926 22:44:16.311338 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:16.311339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:16.311347 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:16.311403 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:16.311986 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:16.312745 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:16.313722 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:16.314052 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:16.314113 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:16.315589 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:16.315590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:16.315591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:16.315597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:16.317131 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:16.317510 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:16.317511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:16.317512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:16.317567 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:16.317761 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:16.317762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:16.317763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:16.317904 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:16.317993 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:16.318134 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 22:44:16.318219 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:44:16.323307 1155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad2066e881e74df1a338e084d7780056: Generated new TSK 0
I20260926 22:44:16.323477 1155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:16.325759 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 1075
2026-09-26T22:44:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:16 dist-test-slave-j8sq krb5kdc[1174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:16 dist-test-slave-j8sq krb5kdc[1174](info): set up 2 sockets
Sep 26 22:44:16 dist-test-slave-j8sq krb5kdc[1174](info): commencing operation
krb5kdc: starting...
W20260926 22:44:17.351052 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:17 dist-test-slave-j8sq krb5kdc[1174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:44:17.464715 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:36113
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:36113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:17.557607 1187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:17.557807 1187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:17.557830 1187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:17.559069 1187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:17.559110 1187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:17.559125 1187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:17.559137 1187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:17.559149 1187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:17.560575 1187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:36113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:36113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:17.560761 1187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:17.560917 1187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:17.565505 1195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:17.565618 1193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:17.565506 1192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:17.565632 1187 server_base.cc:1034] running on GCE node
I20260926 22:44:17.565990 1187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:17.566221 1187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:17.567375 1187 hybrid_clock.cc:648] HybridClock initialized: now 1790462657567374 us; error 35 us; skew 500 ppm
Sep 26 22:44:17 dist-test-slave-j8sq krb5kdc[1174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462657, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:17.569733 1187 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:44:17.570226 1187 webserver.cc:460] Webserver started at http://127.11.107.254:45041/ using document root <none> and password file <none>
I20260926 22:44:17.570498 1187 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:17.570561 1187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:17.570691 1187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:17.571671 1187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "fc29d6430c9e4514b78b8e16bcfbae48"
format_stamp: "Formatted at 2026-09-26 22:44:17 on dist-test-slave-j8sq"
server_key: "d41645e0d0e20b7293e03ba49284010d"
server_key_iv: "c7a473aee15afa1b43de8bc72acac5d2"
server_key_version: "encryptionkey@0"
I20260926 22:44:17.572027 1187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "fc29d6430c9e4514b78b8e16bcfbae48"
format_stamp: "Formatted at 2026-09-26 22:44:17 on dist-test-slave-j8sq"
server_key: "d41645e0d0e20b7293e03ba49284010d"
server_key_iv: "c7a473aee15afa1b43de8bc72acac5d2"
server_key_version: "encryptionkey@0"
I20260926 22:44:17.573354 1187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:44:17.574141 1202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:17.574316 1187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:44:17.574414 1187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "fc29d6430c9e4514b78b8e16bcfbae48"
format_stamp: "Formatted at 2026-09-26 22:44:17 on dist-test-slave-j8sq"
server_key: "d41645e0d0e20b7293e03ba49284010d"
server_key_iv: "c7a473aee15afa1b43de8bc72acac5d2"
server_key_version: "encryptionkey@0"
I20260926 22:44:17.574483 1187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:17.596818 1187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:17.598881 1187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:17.599037 1187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:17.603154 1187 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:36113
I20260926 22:44:17.603192 1254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:36113 every 8 connection(s)
I20260926 22:44:17.603531 1187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:17.604332 1255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:17.606524 1255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48: Bootstrap starting.
I20260926 22:44:17.607081 1255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:17.607345 1255 log.cc:824] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:17.607993 1255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48: No bootstrap required, opened a new log
I20260926 22:44:17.609334 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 1187
I20260926 22:44:17.609452 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:17.609638 11695 external_mini_cluster.cc:1428] Setting key fe3c6fcafac82158b9ca118eb8ae2b27
I20260926 22:44:17.609828 1255 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc29d6430c9e4514b78b8e16bcfbae48" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 36113 } }
I20260926 22:44:17.610015 1255 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:17.610075 1255 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc29d6430c9e4514b78b8e16bcfbae48, State: Initialized, Role: FOLLOWER
I20260926 22:44:17.610224 1255 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc29d6430c9e4514b78b8e16bcfbae48" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 36113 } }
I20260926 22:44:17.610313 1255 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:17.610386 1255 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:17.610440 1255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:17.611251 1255 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc29d6430c9e4514b78b8e16bcfbae48" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 36113 } }
I20260926 22:44:17.611361 1255 leader_election.cc:302] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc29d6430c9e4514b78b8e16bcfbae48; no voters:
I20260926 22:44:17.611524 1255 leader_election.cc:288] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:17.611562 1260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:17.611737 1260 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [term 1 LEADER]: Becoming Leader. State: Replica: fc29d6430c9e4514b78b8e16bcfbae48, State: Running, Role: LEADER
I20260926 22:44:17.611765 1255 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:17.611876 1260 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc29d6430c9e4514b78b8e16bcfbae48" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 36113 } }
I20260926 22:44:17.612231 1260 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc29d6430c9e4514b78b8e16bcfbae48. Latest consensus state: current_term: 1 leader_uuid: "fc29d6430c9e4514b78b8e16bcfbae48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc29d6430c9e4514b78b8e16bcfbae48" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 36113 } } }
I20260926 22:44:17.612390 1261 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc29d6430c9e4514b78b8e16bcfbae48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc29d6430c9e4514b78b8e16bcfbae48" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 36113 } } }
I20260926 22:44:17.612475 1261 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:17.612387 1260 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:17.612808 1267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:17.613278 1267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:17.615159 1267 catalog_manager.cc:1347] Generated new cluster ID: f6617e27e4b8475fae555a5cf052ddb1
I20260926 22:44:17.615211 1267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:17.615633 1259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:17.610268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49513 (local address 127.11.107.254:36113)
0926 22:44:17.610443 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:17.610452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:17.610482 (+ 30us) server_negotiation.cc:407] Connection header received
0926 22:44:17.610602 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:17.610607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:17.610648 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:17.610781 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:17.610976 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:17.612175 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:17.613322 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:17.614060 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:17.614178 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:17.614842 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:17.614846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:17.614847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:17.615041 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:17.615079 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:17.615093 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:17.615163 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:17.615352 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 22:44:17.615415 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260926 22:44:17.616449 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:17.621193 1259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:17.617912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37961 (local address 127.11.107.254:36113)
0926 22:44:17.617943 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:17.617945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:17.618048 (+ 103us) server_negotiation.cc:407] Connection header received
0926 22:44:17.618091 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:17.618092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:17.618102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:17.618170 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:17.618271 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:17.618968 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:17.619897 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:17.620192 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:17.620259 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:17.620954 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:17.620955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:17.620955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:17.620959 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:17.620973 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:17.620977 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:17.621031 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:17.621097 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 22:44:17.621120 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":16}
I20260926 22:44:17.622629 1267 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:17.623104 1267 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:17.633603 1267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc29d6430c9e4514b78b8e16bcfbae48: Generated new TSK 0
I20260926 22:44:17.633782 1267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:17.639391 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 1187
2026-09-26T22:44:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:17 dist-test-slave-j8sq krb5kdc[1286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:17 dist-test-slave-j8sq krb5kdc[1286](info): set up 2 sockets
Sep 26 22:44:17 dist-test-slave-j8sq krb5kdc[1286](info): commencing operation
krb5kdc: starting...
W20260926 22:44:18.671392 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:18 dist-test-slave-j8sq krb5kdc[1286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:44:18.785072 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:39881
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:39881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:18.863817 1299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:18.864048 1299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:18.864085 1299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:18.865204 1299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:18.865270 1299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:18.865296 1299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:18.865329 1299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:18.865351 1299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:18.866881 1299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:39881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:39881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:18.867151 1299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:18.867378 1299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:18.872251 1304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:18.872301 1299 server_base.cc:1034] running on GCE node
W20260926 22:44:18.872284 1305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:18.872277 1307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:18.872678 1299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:18.872926 1299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:18.874095 1299 hybrid_clock.cc:648] HybridClock initialized: now 1790462658874064 us; error 34 us; skew 500 ppm
Sep 26 22:44:18 dist-test-slave-j8sq krb5kdc[1286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462658, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:18.880163 1299 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:44:18.880611 1299 webserver.cc:460] Webserver started at http://127.11.107.254:34655/ using document root <none> and password file <none>
I20260926 22:44:18.880800 1299 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:18.880857 1299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:18.880946 1299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:18.882017 1299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "c79a3c03d9b34ea49db309e00be4c9d7"
format_stamp: "Formatted at 2026-09-26 22:44:18 on dist-test-slave-j8sq"
server_key: "3c35fb7637a3ed0d1ce67eb5173d4275"
server_key_iv: "b1ab20b82443a9aefc0b5b079320e3ef"
server_key_version: "encryptionkey@0"
I20260926 22:44:18.882326 1299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "c79a3c03d9b34ea49db309e00be4c9d7"
format_stamp: "Formatted at 2026-09-26 22:44:18 on dist-test-slave-j8sq"
server_key: "3c35fb7637a3ed0d1ce67eb5173d4275"
server_key_iv: "b1ab20b82443a9aefc0b5b079320e3ef"
server_key_version: "encryptionkey@0"
I20260926 22:44:18.883572 1299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:44:18.884333 1314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:18.884518 1299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:44:18.884604 1299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "c79a3c03d9b34ea49db309e00be4c9d7"
format_stamp: "Formatted at 2026-09-26 22:44:18 on dist-test-slave-j8sq"
server_key: "3c35fb7637a3ed0d1ce67eb5173d4275"
server_key_iv: "b1ab20b82443a9aefc0b5b079320e3ef"
server_key_version: "encryptionkey@0"
I20260926 22:44:18.884653 1299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:18.887748 1299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:18.889564 1299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:18.889667 1299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:18.893464 1299 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:39881
I20260926 22:44:18.893553 1366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:39881 every 8 connection(s)
I20260926 22:44:18.893822 1299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:18.894788 1367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:18.896845 1367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7: Bootstrap starting.
I20260926 22:44:18.897660 1367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:18.898080 1367 log.cc:824] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:18.898758 1367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7: No bootstrap required, opened a new log
I20260926 22:44:18.899299 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 1299
I20260926 22:44:18.899408 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:18.899609 11695 external_mini_cluster.cc:1428] Setting key 161fd15c1d89c72736cc549f3d17685f
I20260926 22:44:18.900800 1367 raft_consensus.cc:348] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79a3c03d9b34ea49db309e00be4c9d7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39881 } }
I20260926 22:44:18.900962 1367 raft_consensus.cc:374] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:44:18.900987 1367 raft_consensus.cc:729] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79a3c03d9b34ea49db309e00be4c9d7, State: Initialized, Role: FOLLOWER
I20260926 22:44:18.901113 1367 consensus_queue.cc:260] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79a3c03d9b34ea49db309e00be4c9d7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39881 } }
I20260926 22:44:18.901188 1367 raft_consensus.cc:388] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:44:18.901212 1367 raft_consensus.cc:482] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:44:18.901254 1367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:18.902175 1367 raft_consensus.cc:504] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79a3c03d9b34ea49db309e00be4c9d7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39881 } }
I20260926 22:44:18.902334 1367 leader_election.cc:302] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c79a3c03d9b34ea49db309e00be4c9d7; no voters:
I20260926 22:44:18.902525 1367 leader_election.cc:288] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:44:18.902751 1373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:18.902854 1367 sys_catalog.cc:564] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:18.902941 1373 raft_consensus.cc:686] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [term 1 LEADER]: Becoming Leader. State: Replica: c79a3c03d9b34ea49db309e00be4c9d7, State: Running, Role: LEADER
I20260926 22:44:18.903128 1373 consensus_queue.cc:237] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79a3c03d9b34ea49db309e00be4c9d7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39881 } }
I20260926 22:44:18.903478 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:18.900103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55575 (local address 127.11.107.254:39881)
0926 22:44:18.901970 (+ 1867us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:18.901978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:18.902020 (+ 42us) server_negotiation.cc:407] Connection header received
0926 22:44:18.902115 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:18.902120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:18.902141 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:18.902278 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:18.902997 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:18.903001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:18.903002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:18.903006 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:18.903049 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:18.903064 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:18.903105 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:18.903249 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 22:44:18.903291 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1820,"thread_start_us":74,"threads_started":1}
I20260926 22:44:18.903638 1374 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c79a3c03d9b34ea49db309e00be4c9d7. Latest consensus state: current_term: 1 leader_uuid: "c79a3c03d9b34ea49db309e00be4c9d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79a3c03d9b34ea49db309e00be4c9d7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39881 } } }
I20260926 22:44:18.903716 1374 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:18.903934 1381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:18.904157 1372 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c79a3c03d9b34ea49db309e00be4c9d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79a3c03d9b34ea49db309e00be4c9d7" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 39881 } } }
I20260926 22:44:18.904251 1372 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:18.904475 1381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:18.905371 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:18.906167 1381 catalog_manager.cc:1347] Generated new cluster ID: e032dcc3c4254dfe93a53d0f1785afa2
I20260926 22:44:18.906222 1381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:18.911358 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:18.910236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48327 (local address 127.11.107.254:39881)
0926 22:44:18.910272 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:18.910274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:18.910375 (+ 101us) server_negotiation.cc:407] Connection header received
0926 22:44:18.910450 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:18.910451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:18.910457 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:18.910513 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:18.911119 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:18.911120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:18.911120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:44:18.911124 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:18.911145 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:44:18.911149 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:18.911199 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:18.911252 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 22:44:18.911265 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:44:18.915647 1381 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:18.915896 1381 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:18.920465 1381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c79a3c03d9b34ea49db309e00be4c9d7: Generated new TSK 0
I20260926 22:44:18.920670 1381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:18.923957 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 1299
2026-09-26T22:44:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1284 ms)
[----------] 18 tests from AuthTokenIssuingTest (23617 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:18 dist-test-slave-j8sq krb5kdc[1398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:18 dist-test-slave-j8sq krb5kdc[1398](info): set up 2 sockets
Sep 26 22:44:18 dist-test-slave-j8sq krb5kdc[1398](info): commencing operation
krb5kdc: starting...
W20260926 22:44:19.946969 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:44:20.064584 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:45093
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:45093,127.11.107.253:41579,127.11.107.252:43807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:20.144443 1411 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 22:44:20.144652 1411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:20.144671 1411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:20.144690 1411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:20.145869 1411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:20.145907 1411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:20.145921 1411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:20.145933 1411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:20.145983 1411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:20.147384 1411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:45093,127.11.107.253:41579,127.11.107.252:43807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:45093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:20.147622 1411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:20.147831 1411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:20.152743 1419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:20.152768 1416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:20.152807 1411 server_base.cc:1034] running on GCE node
W20260926 22:44:20.152716 1417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:20.153156 1411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:20.153403 1411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:20.154557 1411 hybrid_clock.cc:648] HybridClock initialized: now 1790462660154533 us; error 36 us; skew 500 ppm
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:20.157078 1411 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:44:20.157497 1411 webserver.cc:460] Webserver started at http://127.11.107.254:33975/ using document root <none> and password file <none>
I20260926 22:44:20.157752 1411 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:20.157800 1411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:20.157949 1411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:20.159003 1411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "bd13a8784eb748068ed43c4017ff43f9"
format_stamp: "Formatted at 2026-09-26 22:44:20 on dist-test-slave-j8sq"
server_key: "0db526aefb75ce71fb4ec6c65536f56e"
server_key_iv: "a2b9d58a489201a009708abb1b0bc4d6"
server_key_version: "encryptionkey@0"
I20260926 22:44:20.159376 1411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "bd13a8784eb748068ed43c4017ff43f9"
format_stamp: "Formatted at 2026-09-26 22:44:20 on dist-test-slave-j8sq"
server_key: "0db526aefb75ce71fb4ec6c65536f56e"
server_key_iv: "a2b9d58a489201a009708abb1b0bc4d6"
server_key_version: "encryptionkey@0"
I20260926 22:44:20.160733 1411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:44:20.161492 1426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:20.161700 1411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:44:20.161816 1411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "bd13a8784eb748068ed43c4017ff43f9"
format_stamp: "Formatted at 2026-09-26 22:44:20 on dist-test-slave-j8sq"
server_key: "0db526aefb75ce71fb4ec6c65536f56e"
server_key_iv: "a2b9d58a489201a009708abb1b0bc4d6"
server_key_version: "encryptionkey@0"
I20260926 22:44:20.161944 1411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:20.176626 1411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:20.178591 1411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:20.178750 1411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:20.182618 1411 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:45093
I20260926 22:44:20.182663 1478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:45093 every 8 connection(s)
I20260926 22:44:20.183027 1411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:20.183866 1479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:20.185375 1479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:44:20.189275 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 1411
I20260926 22:44:20.189375 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:20.189584 11695 external_mini_cluster.cc:1428] Setting key 279f0c84d15fe45bd164ecec7f1cdf44
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:44:20.195883 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.186255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.11.107.254:45093)
0926 22:44:20.186380 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.186385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.186448 (+ 63us) server_negotiation.cc:407] Connection header received
0926 22:44:20.187035 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.187037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.187063 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.187177 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.187710 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.188521 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.189537 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.189960 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.190048 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.192483 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.192489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.192489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.192495 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.194206 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.194723 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.194726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.194726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.194802 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.195104 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.195105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.195105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.195312 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.195377 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.195627 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.195723 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":41,"threads_started":1}
I20260926 22:44:20.195999 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.186112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45093 (local address 127.0.0.1:59496)
0926 22:44:20.186283 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.186361 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:20.186876 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:20.187181 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:20.187183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:20.187456 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:20.187605 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.187609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.188619 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.188620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.189427 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.189433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.190048 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.190049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.190152 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.190653 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:20.190662 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:20.192391 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:20.194288 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.194296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.194298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.194623 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.194887 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.194888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.194888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.195009 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.195391 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:20.195400 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:20.195478 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 22:44:20.195584 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 22:44:20.195836 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":67,"thread_start_us":59,"threads_started":1}
I20260926 22:44:20.196774 1479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:44:20.197182 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.197042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:41579 (local address 127.0.0.1:34164)
0926 22:44:20.197098 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.197152 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:41579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 22:44:20.197340 1428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.107.253:41579: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.11.107.253@KRBTEST.COM; defaulting to no policy
I20260926 22:44:20.197772 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.197653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:41579 (local address 127.0.0.1:34174)
0926 22:44:20.197716 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.197747 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:41579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 22:44:20.197947 1479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.253:41579: Network error: Client connection negotiation failed: client connection to 127.11.107.253:41579: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.253.keytab.
Entry for principal kudu/127.11.107.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.253.keytab.
I20260926 22:44:20.216473 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.253:41579
--webserver_interface=127.11.107.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.253.keytab
--principal=kudu/127.11.107.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:45093,127.11.107.253:41579,127.11.107.252:43807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:44:20.247198 1479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } attempt: 1
I20260926 22:44:20.247730 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.247576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:41579 (local address 127.0.0.1:34176)
0926 22:44:20.247649 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.247695 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:41579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 22:44:20.248118 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.248017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:41579 (local address 127.0.0.1:34192)
0926 22:44:20.248060 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.248098 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:41579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 22:44:20.248255 1479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.253:41579: Network error: Client connection negotiation failed: client connection to 127.11.107.253:41579: connect: Connection refused (error 111)
W20260926 22:44:20.298691 1486 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 22:44:20.298897 1486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:20.298914 1486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:20.298933 1486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:20.300089 1486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:20.300132 1486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:20.300146 1486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:20.300158 1486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:20.300171 1486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:20.301709 1486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:45093,127.11.107.253:41579,127.11.107.252:43807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.253.keytab
--principal=kudu/127.11.107.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.253:41579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.107.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:20.301983 1486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:20.302217 1486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:20.306866 1494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:20.306852 1492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:20.306869 1491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:20.306883 1486 server_base.cc:1034] running on GCE node
I20260926 22:44:20.307271 1486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:20.307564 1486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:20.308743 1486 hybrid_clock.cc:648] HybridClock initialized: now 1790462660308725 us; error 39 us; skew 500 ppm
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:20.311192 1486 init.cc:377] Logged in from keytab as kudu/127.11.107.253@KRBTEST.COM (short username kudu)
I20260926 22:44:20.311594 1486 webserver.cc:460] Webserver started at http://127.11.107.253:37661/ using document root <none> and password file <none>
I20260926 22:44:20.311815 1486 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:20.311879 1486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:20.312026 1486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:20.313076 1486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/data/instance:
uuid: "99a29e6173ff4f3ebb264c41ba9b9745"
format_stamp: "Formatted at 2026-09-26 22:44:20 on dist-test-slave-j8sq"
server_key: "958dce946f8cb3d2a1d6ea9d12e9e14a"
server_key_iv: "9384998fe7a09e55fbf3a93633cba04e"
server_key_version: "encryptionkey@0"
I20260926 22:44:20.313428 1486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/wal/instance:
uuid: "99a29e6173ff4f3ebb264c41ba9b9745"
format_stamp: "Formatted at 2026-09-26 22:44:20 on dist-test-slave-j8sq"
server_key: "958dce946f8cb3d2a1d6ea9d12e9e14a"
server_key_iv: "9384998fe7a09e55fbf3a93633cba04e"
server_key_version: "encryptionkey@0"
I20260926 22:44:20.314775 1486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:44:20.315464 1501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:20.315625 1486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:44:20.315725 1486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/wal
uuid: "99a29e6173ff4f3ebb264c41ba9b9745"
format_stamp: "Formatted at 2026-09-26 22:44:20 on dist-test-slave-j8sq"
server_key: "958dce946f8cb3d2a1d6ea9d12e9e14a"
server_key_iv: "9384998fe7a09e55fbf3a93633cba04e"
server_key_version: "encryptionkey@0"
I20260926 22:44:20.315796 1486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:20.316427 1479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } attempt: 2
I20260926 22:44:20.316809 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.316673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:41579 (local address 127.0.0.1:34204)
0926 22:44:20.316737 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.316775 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:41579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 22:44:20.317171 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.317070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:41579 (local address 127.0.0.1:34218)
0926 22:44:20.317125 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.317151 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:41579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 22:44:20.317282 1479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.253:41579: Network error: Client connection negotiation failed: client connection to 127.11.107.253:41579: connect: Connection refused (error 111)
I20260926 22:44:20.337319 1486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:20.339327 1486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:20.339486 1486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:20.343664 1486 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.253:41579
I20260926 22:44:20.343712 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.253:41579 every 8 connection(s)
I20260926 22:44:20.344058 1486 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/data/info.pb
I20260926 22:44:20.344877 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:20.346345 1554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:44:20.351136 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 1486
I20260926 22:44:20.351260 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-1/wal/instance
I20260926 22:44:20.351464 11695 external_mini_cluster.cc:1428] Setting key bfa7e4be45a699f88bfcc0b738c3cb60
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.253@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:44:20.355432 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.346978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.11.107.254:45093)
0926 22:44:20.347049 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.347051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.347220 (+ 169us) server_negotiation.cc:407] Connection header received
0926 22:44:20.347673 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.347674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.347684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.347740 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.348328 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.348804 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.349811 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.350052 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.350105 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.352394 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.352396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.352396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.352402 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.354092 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.354474 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.354476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.354476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.354539 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.354768 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.354769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.354770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.354916 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.354983 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.355231 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.355322 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 22:44:20.355559 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.346919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45093 (local address 127.0.0.1:59508)
0926 22:44:20.347082 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.347121 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:20.347575 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:20.347815 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:20.347818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:20.348050 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:20.348253 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.348257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.348881 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.348882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.349739 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.349742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.350119 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.350119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.350236 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.350633 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:20.350641 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:20.352287 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:20.354181 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.354187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.354188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.354400 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.354606 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.354607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.354607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.354705 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.354979 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:20.354983 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:20.355090 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 22:44:20.355182 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 22:44:20.355294 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260926 22:44:20.356141 1554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.107.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.252@KRBTEST.COM" created.
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.253@KRBTEST.COM for kudu/127.11.107.253@KRBTEST.COM
I20260926 22:44:20.366888 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.356409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:41579 (local address 127.0.0.1:34234)
0926 22:44:20.356463 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.356477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:20.356524 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:20.357035 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:20.357036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:20.357225 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:20.357287 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.357289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.358255 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.358257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.359073 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.359075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.359549 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.359550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.359632 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.359980 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:20.359985 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:20.363759 (+ 3774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:20.365908 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.365909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.365909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.366062 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.366273 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.366273 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.366274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.366321 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.366672 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:20.366673 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:20.366707 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 22:44:20.366759 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:44:20.366825 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 22:44:20.367036 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.356444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.11.107.253:41579)
0926 22:44:20.356856 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.356861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.356880 (+ 19us) server_negotiation.cc:407] Connection header received
0926 22:44:20.356909 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.356910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.356922 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.356993 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.357382 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.358173 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.359181 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.359471 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.359523 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.363855 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.363862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.363862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.363868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.365792 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.366141 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.366144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.366144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.366207 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.366382 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.366383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.366383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.366567 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.366694 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.366807 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.366890 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":353,"threads_started":1}
I20260926 22:44:20.367769 1554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:44:20.368116 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.367978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:43807 (local address 127.0.0.1:42738)
0926 22:44:20.368031 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.368084 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 22:44:20.368280 1502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
I20260926 22:44:20.368698 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.368592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:43807 (local address 127.0.0.1:42744)
0926 22:44:20.368642 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.368672 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 22:44:20.368808 1554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:43807: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.252.keytab.
Entry for principal kudu/127.11.107.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.252.keytab.
I20260926 22:44:20.378484 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.252:43807
--webserver_interface=127.11.107.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.252.keytab
--principal=kudu/127.11.107.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:45093,127.11.107.253:41579,127.11.107.252:43807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:44:20.408452 1479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } attempt: 3
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for kudu/127.11.107.253@KRBTEST.COM
I20260926 22:44:20.415793 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.408871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:41579 (local address 127.0.0.1:34236)
0926 22:44:20.408928 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.408949 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:20.409023 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:20.409206 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:20.409207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:20.409383 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:20.409450 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.409451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.410241 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.410242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.410829 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.410831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.411254 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.411256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.411345 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.411843 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:20.411849 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:20.413048 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:20.414784 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.414785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.414785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.414951 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.415151 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.415152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.415152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.415216 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.415468 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:20.415469 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:20.415512 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 22:44:20.415594 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 22:44:20.415692 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 22:44:20.415793 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.408937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.11.107.253:41579)
0926 22:44:20.409011 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.409013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.409025 (+ 12us) server_negotiation.cc:407] Connection header received
0926 22:44:20.409104 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.409105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.409123 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.409200 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.409542 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.410150 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.410920 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.411170 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.411256 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.413139 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.413140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.413141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.413149 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.414686 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.415022 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.415023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.415024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.415080 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.415289 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.415290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.415290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.415397 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.415465 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.415627 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.415699 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 22:44:20.416277 1479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:44:20.416628 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.416533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:43807 (local address 127.0.0.1:42758)
0926 22:44:20.416581 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.416610 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 22:44:20.416949 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.416863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:43807 (local address 127.0.0.1:42760)
0926 22:44:20.416905 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.416929 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 22:44:20.417040 1479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:43807: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
I20260926 22:44:20.418015 1554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } attempt: 1
I20260926 22:44:20.418394 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.418265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:43807 (local address 127.0.0.1:42768)
0926 22:44:20.418334 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.418368 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 22:44:20.418756 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.418668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:43807 (local address 127.0.0.1:42780)
0926 22:44:20.418712 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.418736 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 22:44:20.418866 1554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:43807: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
I20260926 22:44:20.448181 1479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } attempt: 1
I20260926 22:44:20.448661 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.448514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:43807 (local address 127.0.0.1:42786)
0926 22:44:20.448579 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.448614 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 22:44:20.449029 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.448952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:43807 (local address 127.0.0.1:42790)
0926 22:44:20.448990 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.449014 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 22:44:20.449136 1479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:43807: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
W20260926 22:44:20.459534 1561 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 22:44:20.459730 1561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:20.459748 1561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:20.459793 1561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:20.460922 1561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:20.460961 1561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:20.460975 1561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:20.460988 1561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:20.461035 1561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:20.462518 1561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:45093,127.11.107.253:41579,127.11.107.252:43807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.252.keytab
--principal=kudu/127.11.107.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.252:43807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.107.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:20.462769 1561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:20.462983 1561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:20.467686 1566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:20.467700 1567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:20.467758 1561 server_base.cc:1034] running on GCE node
W20260926 22:44:20.467727 1569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:20.468215 1561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:20.468495 1561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:20.469672 1561 hybrid_clock.cc:648] HybridClock initialized: now 1790462660469660 us; error 25 us; skew 500 ppm
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:20.472035 1561 init.cc:377] Logged in from keytab as kudu/127.11.107.252@KRBTEST.COM (short username kudu)
I20260926 22:44:20.472510 1561 webserver.cc:460] Webserver started at http://127.11.107.252:41099/ using document root <none> and password file <none>
I20260926 22:44:20.472748 1561 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:20.472813 1561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:20.472942 1561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:20.474066 1561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/data/instance:
uuid: "b80b64f9981c4ba985ae5838ecae4a30"
format_stamp: "Formatted at 2026-09-26 22:44:20 on dist-test-slave-j8sq"
server_key: "102dbb6b47966fac2cac9a91ef7a1094"
server_key_iv: "36ccad73d9aca945336095fdea6ee9b6"
server_key_version: "encryptionkey@0"
I20260926 22:44:20.474412 1561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/wal/instance:
uuid: "b80b64f9981c4ba985ae5838ecae4a30"
format_stamp: "Formatted at 2026-09-26 22:44:20 on dist-test-slave-j8sq"
server_key: "102dbb6b47966fac2cac9a91ef7a1094"
server_key_iv: "36ccad73d9aca945336095fdea6ee9b6"
server_key_version: "encryptionkey@0"
I20260926 22:44:20.475684 1561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:44:20.476395 1576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:20.476580 1561 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:20.476706 1561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/wal
uuid: "b80b64f9981c4ba985ae5838ecae4a30"
format_stamp: "Formatted at 2026-09-26 22:44:20 on dist-test-slave-j8sq"
server_key: "102dbb6b47966fac2cac9a91ef7a1094"
server_key_iv: "36ccad73d9aca945336095fdea6ee9b6"
server_key_version: "encryptionkey@0"
I20260926 22:44:20.476778 1561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:20.486992 1554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } attempt: 2
I20260926 22:44:20.487375 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.487260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:43807 (local address 127.0.0.1:42804)
0926 22:44:20.487313 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.487348 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 22:44:20.487732 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.487648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:43807 (local address 127.0.0.1:42808)
0926 22:44:20.487696 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.487717 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 22:44:20.487843 1554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:43807: Network error: Client connection negotiation failed: client connection to 127.11.107.252:43807: connect: Connection refused (error 111)
I20260926 22:44:20.493388 1561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:20.495338 1561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:20.495472 1561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:20.499279 1561 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.252:43807
I20260926 22:44:20.499330 1628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.252:43807 every 8 connection(s)
I20260926 22:44:20.499630 1561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/data/info.pb
I20260926 22:44:20.500483 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:20.501967 1629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:44:20.502990 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 1561
I20260926 22:44:20.503119 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462400949426-11695-0/minicluster-data/master-2/wal/instance
I20260926 22:44:20.503304 11695 external_mini_cluster.cc:1428] Setting key 3a0791416dbc45860686b0bbc5503abe
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.252@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.252@KRBTEST.COM
I20260926 22:44:20.511650 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.502606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.11.107.254:45093)
0926 22:44:20.502701 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.502703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.502867 (+ 164us) server_negotiation.cc:407] Connection header received
0926 22:44:20.503491 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.503493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.503505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.503560 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.504394 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.504957 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.505960 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.506144 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.506193 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.508527 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.508528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.508528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.508535 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.510209 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.510745 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.510746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.510746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.510798 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.511057 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.511058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.511058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.511158 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.511206 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.511471 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.511549 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 22:44:20.511857 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.502573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:45093 (local address 127.0.0.1:59522)
0926 22:44:20.502751 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.502783 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:20.503371 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:20.503628 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:20.503632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:20.504005 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:20.504295 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.504301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.505035 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.505036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.505871 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.505874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.506219 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.506220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.506351 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.506867 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:20.506875 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:20.508440 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:20.510278 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.510284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.510286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.510671 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.510865 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.510865 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.510866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.510994 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.511219 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:20.511223 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:20.511350 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 22:44:20.511426 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 22:44:20.511559 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260926 22:44:20.512526 1633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.503716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.11.107.252:43807)
0926 22:44:20.503845 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.503850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.503925 (+ 75us) server_negotiation.cc:407] Connection header received
0926 22:44:20.503976 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.503978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.503995 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.504055 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.504765 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.505605 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.506607 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.507011 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.507064 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.509367 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.509374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.509375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.509381 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.511199 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.511545 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.511548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.511548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.511599 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.511814 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.511817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.511817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.512036 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.512153 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.512289 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.512390 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260926 22:44:20.512676 1629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.252@KRBTEST.COM for kudu/127.11.107.253@KRBTEST.COM
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:44:20.519990 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.512878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:41579 (local address 127.0.0.1:34244)
0926 22:44:20.512984 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.512994 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:20.513025 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:20.513156 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:20.513157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:20.513376 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:20.513431 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.513433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.514161 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.514162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.514760 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.514763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.515100 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.515101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.515180 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.515585 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:20.515591 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:20.516878 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:20.518911 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.518912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.518912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.519083 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.519332 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.519333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.519333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.519400 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.519662 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:20.519663 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:20.519706 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 22:44:20.519809 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 22:44:20.519898 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 22:44:20.519990 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.512951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.11.107.253:41579)
0926 22:44:20.513020 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.513022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.513032 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:44:20.513088 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.513089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.513098 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.513141 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.513508 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.514083 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.514841 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.515031 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.515084 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.516974 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.516975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.516975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.516981 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.518826 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.519173 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.519174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.519174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.519240 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.519462 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.519462 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.519463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.519575 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.519666 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.519837 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.519903 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 22:44:20.520229 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.513205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.11.107.254:45093)
0926 22:44:20.513246 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.513251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.513259 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:44:20.513297 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.513298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.513305 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.513347 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.513822 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.514397 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.515190 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.515420 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.515484 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.517741 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.517742 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.517742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.517749 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.519104 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.519469 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.519470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.519471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.519523 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.519737 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.519738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.519738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.519858 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.520026 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.520060 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.520128 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:44:20.520721 1629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.252@KRBTEST.COM for kudu/127.11.107.252@KRBTEST.COM
I20260926 22:44:20.524262 1479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } attempt: 2
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.253@KRBTEST.COM
I20260926 22:44:20.527829 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.520970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:43807 (local address 127.0.0.1:42836)
0926 22:44:20.521000 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.521009 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:20.521042 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:20.521171 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:20.521172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:20.521279 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:20.521304 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.521305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.522024 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.522025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.522528 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.522530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.522921 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.522922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.522977 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.523497 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:20.523502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:20.525067 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:20.526775 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.526776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.526776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.526931 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.527172 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.527172 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.527173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.527240 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.527499 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:20.527500 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:20.527552 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 22:44:20.527621 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 22:44:20.527728 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 22:44:20.528093 1633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.521000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.11.107.252:43807)
0926 22:44:20.521033 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.521035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.521043 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:44:20.521086 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.521087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.521097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.521170 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.521369 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.521960 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.522600 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.522854 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.522902 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.525150 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.525151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.525151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.525157 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.526685 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.527030 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.527031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.527032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.527089 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.527310 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.527311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.527312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.527423 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.527496 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.527664 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.528014 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":262,"server-negotiator.queue_time_us":18}
I20260926 22:44:20.529166 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.520790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.11.107.253:41579)
0926 22:44:20.520861 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.520863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.520870 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:44:20.520888 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.520889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.520897 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.520977 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.521539 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.522062 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.522686 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.522910 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.522970 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.525303 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.525304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.525304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.525311 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.527767 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.528409 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.528410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.528411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.528466 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.528669 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.528670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.528670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.528784 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.528953 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.529006 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.529073 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for kudu/127.11.107.252@KRBTEST.COM
I20260926 22:44:20.529794 1629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30: Bootstrap starting.
I20260926 22:44:20.530669 1629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:20.530915 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:20.531010 1629 log.cc:824] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:20.531864 1629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30: No bootstrap required, opened a new log
I20260926 22:44:20.532079 1453 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bd13a8784eb748068ed43c4017ff43f9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59532
I20260926 22:44:20.534004 1629 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } }
I20260926 22:44:20.534171 1629 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80b64f9981c4ba985ae5838ecae4a30, State: Initialized, Role: FOLLOWER
I20260926 22:44:20.534281 1629 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } }
I20260926 22:44:20.534368 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.524486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:43807 (local address 127.0.0.1:42850)
0926 22:44:20.524539 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.524549 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:20.524591 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:20.524974 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:20.524975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:20.525127 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:20.525166 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.525168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.526098 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.526099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.526918 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.526921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.527754 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.527755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.527819 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.528221 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:20.528227 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:20.529597 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:20.533257 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.533259 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.533259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.533473 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.533698 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.533699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.533699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.533760 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.534072 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:20.534073 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:20.534114 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 22:44:20.534180 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 22:44:20.534274 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 22:44:20.534469 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.524664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.11.107.252:43807)
0926 22:44:20.524817 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.524818 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.524825 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:44:20.524840 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.524841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.524849 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.525239 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.525290 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.526023 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.526996 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.527692 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.527752 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.529670 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.529671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.529671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.529678 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.533184 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.533556 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.533557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.533557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.533623 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.533827 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.533827 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.533828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.533993 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.534062 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.534232 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.534347 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":112,"threads_started":1}
I20260926 22:44:20.534619 1629 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:20.534592 1636 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } } }
I20260926 22:44:20.534664 1636 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:44:20.536170 1479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9: Bootstrap starting.
W20260926 22:44:20.536435 1647 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:44:20.536526 1647 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:44:20.536859 1479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:20.537154 1479 log.cc:824] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:20.537931 1479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9: No bootstrap required, opened a new log
I20260926 22:44:20.539229 1479 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } }
I20260926 22:44:20.539357 1479 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd13a8784eb748068ed43c4017ff43f9, State: Initialized, Role: FOLLOWER
I20260926 22:44:20.536651 1561 master_runner.cc:186] Error getting master registration for 127.11.107.254:45093: OK, instance_id { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" instance_seqno: 1790462660181445 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 22:44:20.539441 1561 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 22:44:20.539466 1479 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } }
I20260926 22:44:20.539798 1479 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:20.539772 1650 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } } }
I20260926 22:44:20.539844 1650 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:44:20.539906 1453 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 22:44:20.539973 1453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:20.540728 1453 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } }
I20260926 22:44:20.540983 1453 leader_election.cc:288] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [CANDIDATE]: Term 1 election: Requested vote from peers 99a29e6173ff4f3ebb264c41ba9b9745 (127.11.107.253:41579), b80b64f9981c4ba985ae5838ecae4a30 (127.11.107.252:43807)
I20260926 22:44:20.541429 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd13a8784eb748068ed43c4017ff43f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b80b64f9981c4ba985ae5838ecae4a30"
I20260926 22:44:20.541536 1603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:20.542567 1603 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd13a8784eb748068ed43c4017ff43f9 in term 1.
W20260926 22:44:20.541590 1661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:44:20.542976 1661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:44:20.542856 1427 leader_election.cc:302] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b80b64f9981c4ba985ae5838ecae4a30, bd13a8784eb748068ed43c4017ff43f9; no voters:
I20260926 22:44:20.543336 1650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:20.543483 1650 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [term 1 LEADER]: Becoming Leader. State: Replica: bd13a8784eb748068ed43c4017ff43f9, State: Running, Role: LEADER
I20260926 22:44:20.543569 1650 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } }
I20260926 22:44:20.543980 1670 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd13a8784eb748068ed43c4017ff43f9. Latest consensus state: current_term: 1 leader_uuid: "bd13a8784eb748068ed43c4017ff43f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } } }
I20260926 22:44:20.544117 1670 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:20.544442 1411 master_runner.cc:186] Error getting master registration for 127.11.107.253:41579: OK, instance_id { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" instance_seqno: 1790462660342499 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 22:44:20.546646 1411 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 22:44:20.547359 1672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:20.547556 1672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:20.561828 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.542484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.11.107.253:41579)
0926 22:44:20.542738 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.542739 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.554002 (+ 11263us) server_negotiation.cc:407] Connection header received
0926 22:44:20.554064 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.554065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.554077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.554107 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.554843 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.555569 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.556627 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.556959 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.557024 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.558608 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.558610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.558610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.558617 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.560574 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.561052 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.561053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.561053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.561104 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.561257 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.561258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.561258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.561396 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.561488 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.561601 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.561699 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234}
I20260926 22:44:20.562207 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.542332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.11.107.252:43807)
0926 22:44:20.542403 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.542405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.547201 (+ 4796us) server_negotiation.cc:407] Connection header received
0926 22:44:20.547273 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.547274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.547302 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.547350 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.549106 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.549934 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.552303 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.552654 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.552717 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.555391 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.555392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.555392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.555399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.557055 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.560251 (+ 3196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.560253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.560253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.560309 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.560513 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.560514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.560514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.560647 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.560747 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.562007 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.562092 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 22:44:20.563122 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.542365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.11.107.254:45093)
0926 22:44:20.542700 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.542703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.548319 (+ 5616us) server_negotiation.cc:407] Connection header received
0926 22:44:20.548410 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.548411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.548420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.548467 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.549737 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.550424 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.551438 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.551729 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.551792 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.557559 (+ 5767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.557561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.557561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.557571 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.559834 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.561295 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.561297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.561297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.561345 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.562501 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.562502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.562502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.562663 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.562737 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.562951 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.563029 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260926 22:44:20.563822 1603 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30 [term 1 FOLLOWER]: Refusing update from remote peer bd13a8784eb748068ed43c4017ff43f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:44:20.564162 1650 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:44:20.565564 1636 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd13a8784eb748068ed43c4017ff43f9. Latest consensus state: current_term: 1 leader_uuid: "bd13a8784eb748068ed43c4017ff43f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } } }
I20260926 22:44:20.565662 1636 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:44:20.567270 1650 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd13a8784eb748068ed43c4017ff43f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } } }
I20260926 22:44:20.567363 1650 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:20.567379 1672 catalog_manager.cc:1347] Generated new cluster ID: 3cdcd05a0b8646e99e4190df57713ade
I20260926 22:44:20.567420 1672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:20.567893 1636 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd13a8784eb748068ed43c4017ff43f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } } }
I20260926 22:44:20.567979 1636 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:44:20.569952 1674 mvcc.cc:205] Tried to move back new op lower bound from 7333735057626103808 to 7333735057587015680. Current Snapshot: MvccSnapshot[applied={T|T < 7333735057626103808}]
I20260926 22:44:20.578998 1554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } attempt: 3
Sep 26 22:44:20 dist-test-slave-j8sq krb5kdc[1398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462660, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.253@KRBTEST.COM for kudu/127.11.107.252@KRBTEST.COM
I20260926 22:44:20.585096 1672 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:20.585930 1672 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:20.588559 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.579274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:43807 (local address 127.0.0.1:42874)
0926 22:44:20.579344 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:20.579354 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:20.579386 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:20.579703 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:20.579704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:20.579883 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:20.579946 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.579948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.581016 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.581017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.581749 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:20.581751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.582417 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.582418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:20.582486 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.582984 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:20.582989 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:20.584673 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:20.587402 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.587403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.587404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.587593 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.587867 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:20.587868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:20.587868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:20.587929 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:20.588284 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:20.588285 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:20.588337 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 22:44:20.588381 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 22:44:20.588464 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 22:44:20.589051 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:20.579458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.11.107.252:43807)
0926 22:44:20.579562 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.579564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.579571 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:44:20.579604 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.579604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.579613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.580064 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.580106 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.580916 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.582016 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:20.582325 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:20.584628 (+ 2303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:20.584847 (+ 219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:20.584848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:20.584848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:20.584854 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:20.587280 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.587675 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.587676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.587677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.587781 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:20.587999 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:20.588000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:20.588000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:20.588199 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:20.588726 (+ 527us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:20.588793 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 22:44:20.588912 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 22:44:20.590440 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745: Bootstrap starting.
I20260926 22:44:20.591104 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:20.591432 1554 log.cc:824] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:20.592558 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745: No bootstrap required, opened a new log
I20260926 22:44:20.594383 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } }
I20260926 22:44:20.594508 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a29e6173ff4f3ebb264c41ba9b9745, State: Initialized, Role: FOLLOWER
I20260926 22:44:20.594579 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } }
I20260926 22:44:20.594854 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:20.595304 1528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd13a8784eb748068ed43c4017ff43f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "99a29e6173ff4f3ebb264c41ba9b9745"
I20260926 22:44:20.595407 1527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:20.596311 1527 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745 [term 1 FOLLOWER]: Refusing update from remote peer bd13a8784eb748068ed43c4017ff43f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:44:20.596589 1670 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:44:20.597100 1528 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd13a8784eb748068ed43c4017ff43f9 in term 1.
I20260926 22:44:20.599532 1694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd13a8784eb748068ed43c4017ff43f9. Latest consensus state: current_term: 1 leader_uuid: "bd13a8784eb748068ed43c4017ff43f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } } }
I20260926 22:44:20.599629 1694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:44:20.599701 1694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd13a8784eb748068ed43c4017ff43f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } } }
I20260926 22:44:20.599759 1694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 22:44:20.600275 1695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:44:20.600333 1695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:44:20.600827 1687 mvcc.cc:205] Tried to move back new op lower bound from 7333735057748049920 to 7333735057587015680. Current Snapshot: MvccSnapshot[applied={T|T < 7333735057626103808}]
I20260926 22:44:20.602166 1677 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd13a8784eb748068ed43c4017ff43f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } } }
I20260926 22:44:20.602278 1677 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:20.602286 1672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd13a8784eb748068ed43c4017ff43f9: Generated new TSK 0
I20260926 22:44:20.602468 1672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:44:20.598567 1680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd13a8784eb748068ed43c4017ff43f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd13a8784eb748068ed43c4017ff43f9" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 45093 } } peers { permanent_uuid: "99a29e6173ff4f3ebb264c41ba9b9745" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 41579 } } peers { permanent_uuid: "b80b64f9981c4ba985ae5838ecae4a30" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 43807 } } }
I20260926 22:44:20.602985 1680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 22:44:20.626502 1482 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:20.624955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.11.107.254:45093)
0926 22:44:20.624996 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.624999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.625832 (+ 833us) server_negotiation.cc:407] Connection header received
0926 22:44:20.625959 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.625960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.625971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.626001 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.626433 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59554: BlockingRecv error: recv got EOF from 127.0.0.1:59554 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 22:44:20.627202 1558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:20.625311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.11.107.253:41579)
0926 22:44:20.625347 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.625349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.626725 (+ 1376us) server_negotiation.cc:407] Connection header received
0926 22:44:20.626819 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.626820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.626830 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.626858 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.627132 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34274: BlockingRecv error: recv got EOF from 127.0.0.1:34274 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 22:44:20.627866 1634 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:20.625594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42888 (local address 127.11.107.252:43807)
0926 22:44:20.625630 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.625633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.627399 (+ 1766us) server_negotiation.cc:407] Connection header received
0926 22:44:20.627489 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.627490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.627500 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.627528 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.627810 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42888: BlockingRecv error: recv got EOF from 127.0.0.1:42888 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 22:44:20.630067 1558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:20.629356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.11.107.253:41579)
0926 22:44:20.629388 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.629389 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.629563 (+ 174us) server_negotiation.cc:407] Connection header received
0926 22:44:20.629663 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.629664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.629680 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.629709 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.630022 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34290: BlockingRecv error: recv got EOF from 127.0.0.1:34290 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:44:20.632925 1558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:20.632213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.11.107.253:41579)
0926 22:44:20.632244 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.632245 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.632480 (+ 235us) server_negotiation.cc:407] Connection header received
0926 22:44:20.632569 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.632570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.632579 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.632606 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.632883 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34300: BlockingRecv error: recv got EOF from 127.0.0.1:34300 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 22:44:20.636704 1558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:20.636000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.11.107.253:41579)
0926 22:44:20.636053 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.636055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.636265 (+ 210us) server_negotiation.cc:407] Connection header received
0926 22:44:20.636352 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.636353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.636362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.636389 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.636663 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34314: BlockingRecv error: recv got EOF from 127.0.0.1:34314 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 22:44:20.642812 1558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:20.642088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.11.107.253:41579)
0926 22:44:20.642121 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.642122 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.642350 (+ 228us) server_negotiation.cc:407] Connection header received
0926 22:44:20.642441 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.642442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.642451 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.642478 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.642769 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34326: BlockingRecv error: recv got EOF from 127.0.0.1:34326 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:44:20.652711 1558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:20.651918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.11.107.253:41579)
0926 22:44:20.651953 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.651955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.652181 (+ 226us) server_negotiation.cc:407] Connection header received
0926 22:44:20.652287 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.652288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.652298 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.652342 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.652652 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34336: BlockingRecv error: recv got EOF from 127.0.0.1:34336 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:44:20.670734 1558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:20.669915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.11.107.253:41579)
0926 22:44:20.669954 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.669956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.670210 (+ 254us) server_negotiation.cc:407] Connection header received
0926 22:44:20.670309 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.670310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.670320 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.670349 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.670680 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34348: BlockingRecv error: recv got EOF from 127.0.0.1:34348 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 22:44:20.705066 1558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:20.704103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.11.107.253:41579)
0926 22:44:20.704164 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.704166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.704460 (+ 294us) server_negotiation.cc:407] Connection header received
0926 22:44:20.704571 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.704572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.704584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.704616 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.705005 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34350: BlockingRecv error: recv got EOF from 127.0.0.1:34350 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 22:44:20.771340 1558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:20.770526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.11.107.253:41579)
0926 22:44:20.770619 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.770621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.770686 (+ 65us) server_negotiation.cc:407] Connection header received
0926 22:44:20.770757 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.770758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.770768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.770818 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.771243 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34354: BlockingRecv error: recv got EOF from 127.0.0.1:34354 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 22:44:20.901312 1558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:20.900568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.11.107.253:41579)
0926 22:44:20.900657 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:20.900659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:20.900742 (+ 83us) server_negotiation.cc:407] Connection header received
0926 22:44:20.900820 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:20.900821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:20.900833 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:20.900884 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:20.901222 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34364: BlockingRecv error: recv got EOF from 127.0.0.1:34364 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 22:44:21.159205 1558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:21.158533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.11.107.253:41579)
0926 22:44:21.158623 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:21.158626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:21.158646 (+ 20us) server_negotiation.cc:407] Connection header received
0926 22:44:21.158726 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:21.158727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:21.158741 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:21.158782 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:21.159143 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34370: BlockingRecv error: recv got EOF from 127.0.0.1:34370 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 22:44:21.537002 1647 catalog_manager.cc:1259] Loaded cluster ID: 3cdcd05a0b8646e99e4190df57713ade
I20260926 22:44:21.537118 1647 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30: loading cluster ID for follower catalog manager: success
I20260926 22:44:21.537977 1647 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30: acquiring CA information for follower catalog manager: success
I20260926 22:44:21.538290 1647 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b80b64f9981c4ba985ae5838ecae4a30: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:44:21.600767 1695 catalog_manager.cc:1259] Loaded cluster ID: 3cdcd05a0b8646e99e4190df57713ade
I20260926 22:44:21.600869 1695 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745: loading cluster ID for follower catalog manager: success
I20260926 22:44:21.601687 1695 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745: acquiring CA information for follower catalog manager: success
I20260926 22:44:21.602032 1695 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99a29e6173ff4f3ebb264c41ba9b9745: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:44:21.676506 1764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:21.672395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.11.107.253:41579)
0926 22:44:21.672526 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:21.672528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:21.672664 (+ 136us) server_negotiation.cc:407] Connection header received
0926 22:44:21.672772 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:21.672773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:21.672864 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:21.672975 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:44:21.673167 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:21.674113 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:21.675466 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:21.675845 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:21.675908 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:21.676098 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:44:21.676241 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:44:21.676338 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:21.676385 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 22:44:21.676409 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260926 22:44:21.681715 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:21.678396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.11.107.252:43807)
0926 22:44:21.678525 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:21.678527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:21.678559 (+ 32us) server_negotiation.cc:407] Connection header received
0926 22:44:21.678662 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:21.678663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:21.678712 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:21.678799 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:44:21.678979 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:21.679714 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:21.680907 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:21.681133 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:21.681216 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:21.681377 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:44:21.681480 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:44:21.681562 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:21.681606 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 22:44:21.681626 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":50,"threads_started":1}
I20260926 22:44:21.685873 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:21.683345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.11.107.252:43807)
0926 22:44:21.683381 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:21.683382 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:21.683441 (+ 59us) server_negotiation.cc:407] Connection header received
0926 22:44:21.683514 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:21.683515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:21.683524 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:21.683570 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:44:21.683708 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:21.684303 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:21.685203 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:21.685405 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:21.685475 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:21.685625 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:44:21.685652 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:44:21.685693 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:21.685752 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 22:44:21.685769 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:44:21.686301 1764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:21.683583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34396 (local address 127.11.107.253:41579)
0926 22:44:21.683621 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:21.683623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:21.683737 (+ 114us) server_negotiation.cc:407] Connection header received
0926 22:44:21.683833 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:21.683834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:21.683843 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:21.683870 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:44:21.684028 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:21.684480 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:21.685501 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:21.685791 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:21.685826 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:21.686032 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:44:21.686063 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:44:21.686092 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:21.686216 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 22:44:21.686241 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:44:21.686798 1776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:21.683275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59566 (local address 127.11.107.254:45093)
0926 22:44:21.683398 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:21.683400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:21.683529 (+ 129us) server_negotiation.cc:407] Connection header received
0926 22:44:21.683651 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:21.683653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:21.683709 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:21.683768 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:44:21.683985 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:21.684725 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:21.685683 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:21.686023 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:21.686061 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:21.686309 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:44:21.686411 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:44:21.686471 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:21.686714 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 22:44:21.686737 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260926 22:44:21.688337 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 1411
I20260926 22:44:21.694005 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 1486
I20260926 22:44:21.698879 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 1561
2026-09-26T22:44:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2777 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:44:21 dist-test-slave-j8sq krb5kdc[1781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:44:21 dist-test-slave-j8sq krb5kdc[1781](info): set up 2 sockets
Sep 26 22:44:21 dist-test-slave-j8sq krb5kdc[1781](info): commencing operation
krb5kdc: starting...
W20260926 22:44:22.735077 11695 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/test-user.keytab.
Sep 26 22:44:22 dist-test-slave-j8sq krb5kdc[1781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:44:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.107.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
Entry for principal kudu/127.11.107.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab.
I20260926 22:44:22.850004 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:35635
--webserver_interface=127.11.107.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:34787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:35635,127.11.107.253:33041,127.11.107.252:44279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:44:22.929653 1794 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 22:44:22.929906 1794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:22.929949 1794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:22.929986 1794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:22.931141 1794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:22.931205 1794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:22.931229 1794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:22.931258 1794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:22.931288 1794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:22.932739 1794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:34787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:35635,127.11.107.253:33041,127.11.107.252:44279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.254.keytab
--principal=kudu/127.11.107.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.254:35635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.107.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:22.932999 1794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:22.933215 1794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:22.937956 1799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:22.937943 1802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:22.937970 1800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:22.938407 1794 server_base.cc:1034] running on GCE node
I20260926 22:44:22.938655 1794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:22.938944 1794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:22.940138 1794 hybrid_clock.cc:648] HybridClock initialized: now 1790462662940112 us; error 38 us; skew 500 ppm
Sep 26 22:44:22 dist-test-slave-j8sq krb5kdc[1781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462662, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:22.942711 1794 init.cc:377] Logged in from keytab as kudu/127.11.107.254@KRBTEST.COM (short username kudu)
I20260926 22:44:22.943181 1794 webserver.cc:460] Webserver started at http://127.11.107.254:42055/ using document root <none> and password file <none>
I20260926 22:44:22.943404 1794 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:22.943470 1794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:22.943601 1794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:22.944667 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/data/instance:
uuid: "79701f22f64848a4ae0708896724d4be"
format_stamp: "Formatted at 2026-09-26 22:44:22 on dist-test-slave-j8sq"
server_key: "47e0c22ad720e2b2068bcc3e495d692c"
server_key_iv: "523886fe95a022b9c53eb7c35e7e6ede"
server_key_version: "encryptionkey@0"
I20260926 22:44:22.945024 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/wal/instance:
uuid: "79701f22f64848a4ae0708896724d4be"
format_stamp: "Formatted at 2026-09-26 22:44:22 on dist-test-slave-j8sq"
server_key: "47e0c22ad720e2b2068bcc3e495d692c"
server_key_iv: "523886fe95a022b9c53eb7c35e7e6ede"
server_key_version: "encryptionkey@0"
I20260926 22:44:22.946398 1794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:44:22.947118 1809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:22.947296 1794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:22.947391 1794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/wal
uuid: "79701f22f64848a4ae0708896724d4be"
format_stamp: "Formatted at 2026-09-26 22:44:22 on dist-test-slave-j8sq"
server_key: "47e0c22ad720e2b2068bcc3e495d692c"
server_key_iv: "523886fe95a022b9c53eb7c35e7e6ede"
server_key_version: "encryptionkey@0"
I20260926 22:44:22.947469 1794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:22.957165 1794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:22.959388 1794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:22.959542 1794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:22.963496 1794 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.254:35635
I20260926 22:44:22.963526 1861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.254:35635 every 8 connection(s)
I20260926 22:44:22.963871 1794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/data/info.pb
I20260926 22:44:22.964287 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 1794
I20260926 22:44:22.964414 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-0/wal/instance
I20260926 22:44:22.964602 11695 external_mini_cluster.cc:1428] Setting key 6dcae800fd0ac8982ca1e61463774306
I20260926 22:44:22.964808 1862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:22.966388 1862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.107.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.253@KRBTEST.COM" created.
Sep 26 22:44:22 dist-test-slave-j8sq krb5kdc[1781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462662, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:44:22.980206 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:22.967050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:35635 (local address 127.0.0.1:41176)
0926 22:44:22.967231 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:22.967277 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:22.967958 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:22.968339 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:22.968342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:22.968704 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:22.968925 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:22.968930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:22.970012 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:22.970014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:22.971176 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:22.971178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:22.971684 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:22.971685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:22.971773 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:22.972298 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:22.972309 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:22.976476 (+ 4167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:22.978728 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:22.978735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:22.978736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:22.978939 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:22.979228 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:22.979229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:22.979230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:22.979320 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:22.979776 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:22.979781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:22.979839 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 22:44:22.979915 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 22:44:22.980042 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260926 22:44:22.980319 1866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:22.967149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.11.107.254:35635)
0926 22:44:22.967285 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:22.967289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:22.967354 (+ 65us) server_negotiation.cc:407] Connection header received
0926 22:44:22.968144 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:22.968146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:22.968164 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:22.968309 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:22.969026 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:22.969914 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:22.971289 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:22.971585 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:22.971669 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:22.976568 (+ 4899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:22.976575 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:22.976575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:22.976581 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:22.978640 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:22.979028 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:22.979031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:22.979031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:22.979143 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:22.979390 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:22.979391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:22.979392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:22.979699 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:22.979793 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:22.979974 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 22:44:22.980133 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260926 22:44:22.982113 1862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:44:22.982465 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:22.982352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:33041 (local address 127.0.0.1:40478)
0926 22:44:22.982401 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:22.982445 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:33041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 22:44:22.982650 1813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.107.253:33041: connect: Connection refused (error 111)
I20260926 22:44:22.983072 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:22.982981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:33041 (local address 127.0.0.1:40486)
0926 22:44:22.983028 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:22.983054 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:33041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 22:44:22.983194 1862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.253:33041: Network error: Client connection negotiation failed: client connection to 127.11.107.253:33041: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.253.keytab.
Entry for principal kudu/127.11.107.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.253.keytab.
I20260926 22:44:22.990801 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.253:33041
--webserver_interface=127.11.107.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.253.keytab
--principal=kudu/127.11.107.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:34787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:35635,127.11.107.253:33041,127.11.107.252:44279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:44:23.032433 1862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } attempt: 1
I20260926 22:44:23.032930 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.032790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:33041 (local address 127.0.0.1:40500)
0926 22:44:23.032861 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.032899 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:33041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 22:44:23.033290 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.033195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:33041 (local address 127.0.0.1:40502)
0926 22:44:23.033242 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.033271 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:33041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 22:44:23.033376 1862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.253:33041: Network error: Client connection negotiation failed: client connection to 127.11.107.253:33041: connect: Connection refused (error 111)
W20260926 22:44:23.071214 1869 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 22:44:23.071415 1869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:23.071432 1869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:23.071452 1869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:23.072630 1869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:23.072669 1869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:23.072685 1869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:23.072721 1869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:23.072748 1869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:23.074249 1869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:34787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:35635,127.11.107.253:33041,127.11.107.252:44279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.253.keytab
--principal=kudu/127.11.107.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.253:33041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.107.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:23.074506 1869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:23.074728 1869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:44:23.079461 1877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:23.079542 1874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:23.079571 1875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:23.079473 1869 server_base.cc:1034] running on GCE node
I20260926 22:44:23.079910 1869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:23.080144 1869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:23.081283 1869 hybrid_clock.cc:648] HybridClock initialized: now 1790462663081266 us; error 32 us; skew 500 ppm
Sep 26 22:44:23 dist-test-slave-j8sq krb5kdc[1781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462663, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:23.083748 1869 init.cc:377] Logged in from keytab as kudu/127.11.107.253@KRBTEST.COM (short username kudu)
I20260926 22:44:23.084211 1869 webserver.cc:460] Webserver started at http://127.11.107.253:44571/ using document root <none> and password file <none>
I20260926 22:44:23.084450 1869 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:23.084517 1869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:23.084676 1869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:23.085705 1869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/data/instance:
uuid: "98ecfbbb31c649c7bb0e1bec7160cb57"
format_stamp: "Formatted at 2026-09-26 22:44:23 on dist-test-slave-j8sq"
server_key: "8a253d9091f1de67525733241f5b300f"
server_key_iv: "b336c281e78db1bc48db6b41892d6685"
server_key_version: "encryptionkey@0"
I20260926 22:44:23.086087 1869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/wal/instance:
uuid: "98ecfbbb31c649c7bb0e1bec7160cb57"
format_stamp: "Formatted at 2026-09-26 22:44:23 on dist-test-slave-j8sq"
server_key: "8a253d9091f1de67525733241f5b300f"
server_key_iv: "b336c281e78db1bc48db6b41892d6685"
server_key_version: "encryptionkey@0"
I20260926 22:44:23.087423 1869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:44:23.088181 1884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:23.088366 1869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:44:23.088464 1869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/wal
uuid: "98ecfbbb31c649c7bb0e1bec7160cb57"
format_stamp: "Formatted at 2026-09-26 22:44:23 on dist-test-slave-j8sq"
server_key: "8a253d9091f1de67525733241f5b300f"
server_key_iv: "b336c281e78db1bc48db6b41892d6685"
server_key_version: "encryptionkey@0"
I20260926 22:44:23.088541 1869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:23.101039 1869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:23.101529 1862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } attempt: 2
I20260926 22:44:23.101972 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.101806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:33041 (local address 127.0.0.1:40518)
0926 22:44:23.101903 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.101940 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:33041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 22:44:23.102408 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.102294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:33041 (local address 127.0.0.1:40522)
0926 22:44:23.102348 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.102378 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.253:33041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 22:44:23.102526 1862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.253:33041: Network error: Client connection negotiation failed: client connection to 127.11.107.253:33041: connect: Connection refused (error 111)
I20260926 22:44:23.103398 1869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:23.103598 1869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:23.107899 1869 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.253:33041
I20260926 22:44:23.107960 1936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.253:33041 every 8 connection(s)
I20260926 22:44:23.108422 1869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/data/info.pb
I20260926 22:44:23.109382 1937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:23.110986 1937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:44:23.115175 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 1869
I20260926 22:44:23.115294 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-1/wal/instance
I20260926 22:44:23.115509 11695 external_mini_cluster.cc:1428] Setting key a00f17babbdbf44d787d190e35711a25
Sep 26 22:44:23 dist-test-slave-j8sq krb5kdc[1781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462663, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.253@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:44:23.120324 1866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.111691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.11.107.254:35635)
0926 22:44:23.111756 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.111758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.111936 (+ 178us) server_negotiation.cc:407] Connection header received
0926 22:44:23.112408 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.112409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.112421 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.112472 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.113031 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.113516 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.114538 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.114787 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.114857 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.117293 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.117295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.117295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.117303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.118880 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.119324 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.119325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.119326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.119395 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.119652 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.119653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.119654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.119805 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.119886 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.120154 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.120238 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:44:23.120499 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.111661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:35635 (local address 127.0.0.1:41188)
0926 22:44:23.111817 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.111856 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:23.112293 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:23.112532 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:23.112535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:23.112760 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:23.112939 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.112942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.113628 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.113629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.114443 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.114446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.114889 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.114890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.114992 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.115493 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:23.115501 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:23.117182 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:23.118985 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.118993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.118995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.119231 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.119466 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.119467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.119467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.119582 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.119878 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:23.119884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:23.120018 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 22:44:23.120133 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 22:44:23.120278 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260926 22:44:23.121191 1937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.107.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.107.252@KRBTEST.COM" created.
Sep 26 22:44:23 dist-test-slave-j8sq krb5kdc[1781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462663, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.253@KRBTEST.COM for kudu/127.11.107.253@KRBTEST.COM
I20260926 22:44:23.129117 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.121472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:33041 (local address 127.0.0.1:40534)
0926 22:44:23.121675 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.121685 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:23.121737 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:23.121929 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:23.121930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:23.122089 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:23.122170 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.122171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.123073 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.123074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.123630 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.123632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.124126 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.124127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.124185 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.124567 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:23.124571 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:23.126194 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:23.127895 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.127896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.127897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.128119 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.128364 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.128365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.128366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.128435 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.128729 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:23.128730 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:23.128785 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 22:44:23.128869 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 22:44:23.128976 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260926 22:44:23.129262 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.121554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.11.107.253:33041)
0926 22:44:23.121692 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.121696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.121733 (+ 37us) server_negotiation.cc:407] Connection header received
0926 22:44:23.121796 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.121797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.121809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.121932 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.122255 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.122928 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.123703 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.124032 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.124107 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.126272 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.126280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.126280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.126287 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.127813 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.128221 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.128224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.128224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.128285 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.128500 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.128501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.128501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.128658 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.128726 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.128893 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.129066 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260926 22:44:23.130007 1937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:44:23.130349 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.130224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:44279 (local address 127.0.0.1:55674)
0926 22:44:23.130272 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.130318 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 22:44:23.130491 1886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
I20260926 22:44:23.130976 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.130855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:44279 (local address 127.0.0.1:55688)
0926 22:44:23.130902 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.130946 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 22:44:23.131093 1937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:44279: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.107.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.252.keytab.
Entry for principal kudu/127.11.107.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.252.keytab.
I20260926 22:44:23.140116 11695 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4c2jnj/build/release/bin/kudu
/tmp/dist-test-task4c2jnj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.252:44279
--webserver_interface=127.11.107.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.252.keytab
--principal=kudu/127.11.107.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.107.212:34787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.107.254:35635,127.11.107.253:33041,127.11.107.252:44279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:44:23.180320 1937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } attempt: 1
I20260926 22:44:23.180948 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.180808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:44279 (local address 127.0.0.1:55704)
0926 22:44:23.180879 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.180917 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 22:44:23.181365 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.181273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:44279 (local address 127.0.0.1:55718)
0926 22:44:23.181319 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.181345 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 22:44:23.181481 1937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:44279: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
I20260926 22:44:23.193674 1862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } attempt: 3
Sep 26 22:44:23 dist-test-slave-j8sq krb5kdc[1781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462662, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for kudu/127.11.107.253@KRBTEST.COM
I20260926 22:44:23.200588 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.194000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:33041 (local address 127.0.0.1:40540)
0926 22:44:23.194057 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.194066 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:23.194123 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:23.194280 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:23.194281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:23.194442 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:23.194506 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.194507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.195213 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.195214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.195748 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.195750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.196126 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.196127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.196209 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.196707 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:23.196712 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:23.197915 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:23.199556 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.199557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.199557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.199747 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.199952 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.199953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.199953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.200012 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.200261 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:23.200262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:23.200309 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 22:44:23.200392 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 22:44:23.200493 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 22:44:23.200587 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.194064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.11.107.253:33041)
0926 22:44:23.194122 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.194125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.194163 (+ 38us) server_negotiation.cc:407] Connection header received
0926 22:44:23.194194 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.194195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.194205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.194260 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.194584 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.195075 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.195833 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.196051 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.196128 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.197998 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.198000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.198000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.198009 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.199415 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.199824 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.199825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.199826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.199885 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.200079 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.200080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.200080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.200194 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.200269 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.200415 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.200493 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:44:23.200999 1862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:44:23.201367 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.201267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:44279 (local address 127.0.0.1:55722)
0926 22:44:23.201319 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.201348 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 22:44:23.201723 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.201634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:44279 (local address 127.0.0.1:55724)
0926 22:44:23.201679 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.201702 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 22:44:23.201812 1862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:44279: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
W20260926 22:44:23.220352 1944 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 22:44:23.220573 1944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:44:23.220602 1944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:44:23.220639 1944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:44:23.221757 1944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:44:23.221815 1944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:44:23.221837 1944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:44:23.221874 1944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:44:23.221902 1944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:44:23.223316 1944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.107.212:34787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.107.254:35635,127.11.107.253:33041,127.11.107.252:44279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/krb5kdc/kudu_127.11.107.252.keytab
--principal=kudu/127.11.107.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.107.252:44279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.107.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:44:23.223572 1944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:44:23.223788 1944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:44:23.228595 1944 server_base.cc:1034] running on GCE node
W20260926 22:44:23.228554 1952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:23.228716 1950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:44:23.228561 1949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:44:23.228996 1944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:44:23.229331 1944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:44:23.230516 1944 hybrid_clock.cc:648] HybridClock initialized: now 1790462663230509 us; error 39 us; skew 500 ppm
Sep 26 22:44:23 dist-test-slave-j8sq krb5kdc[1781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462663, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:44:23.232976 1862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } attempt: 1
I20260926 22:44:23.233217 1944 init.cc:377] Logged in from keytab as kudu/127.11.107.252@KRBTEST.COM (short username kudu)
I20260926 22:44:23.233446 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.233274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:44279 (local address 127.0.0.1:55730)
0926 22:44:23.233367 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.233409 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 22:44:23.233776 1944 webserver.cc:460] Webserver started at http://127.11.107.252:33697/ using document root <none> and password file <none>
I20260926 22:44:23.233862 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.233730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:44279 (local address 127.0.0.1:55734)
0926 22:44:23.233783 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.233816 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 22:44:23.233961 1862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:44279: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
I20260926 22:44:23.234062 1944 fs_manager.cc:359] Metadata directory not provided
I20260926 22:44:23.234123 1944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:44:23.234267 1944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:44:23.235309 1944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/data/instance:
uuid: "e81d70adea33483da821540ed3f93ebb"
format_stamp: "Formatted at 2026-09-26 22:44:23 on dist-test-slave-j8sq"
server_key: "30985eda5afd92bb7371d53c07a5f630"
server_key_iv: "bfc74e5985625a5c5bdb56393c6b92f6"
server_key_version: "encryptionkey@0"
I20260926 22:44:23.235875 1944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/wal/instance:
uuid: "e81d70adea33483da821540ed3f93ebb"
format_stamp: "Formatted at 2026-09-26 22:44:23 on dist-test-slave-j8sq"
server_key: "30985eda5afd92bb7371d53c07a5f630"
server_key_iv: "bfc74e5985625a5c5bdb56393c6b92f6"
server_key_version: "encryptionkey@0"
I20260926 22:44:23.237530 1944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:44:23.238344 1959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:23.238499 1944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:44:23.238587 1944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/data,/tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/wal
uuid: "e81d70adea33483da821540ed3f93ebb"
format_stamp: "Formatted at 2026-09-26 22:44:23 on dist-test-slave-j8sq"
server_key: "30985eda5afd92bb7371d53c07a5f630"
server_key_iv: "bfc74e5985625a5c5bdb56393c6b92f6"
server_key_version: "encryptionkey@0"
I20260926 22:44:23.238690 1944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:44:23.248428 1944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:44:23.249676 1937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } attempt: 2
I20260926 22:44:23.250072 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.249946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:44279 (local address 127.0.0.1:55746)
0926 22:44:23.250003 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.250035 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 22:44:23.250474 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.250358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:44279 (local address 127.0.0.1:55750)
0926 22:44:23.250408 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.250446 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 22:44:23.250567 1937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.107.252:44279: Network error: Client connection negotiation failed: client connection to 127.11.107.252:44279: connect: Connection refused (error 111)
I20260926 22:44:23.250881 1944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:44:23.251030 1944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:44:23.254980 1944 rpc_server.cc:307] RPC server started. Bound to: 127.11.107.252:44279
I20260926 22:44:23.255002 2011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.107.252:44279 every 8 connection(s)
I20260926 22:44:23.255347 1944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/data/info.pb
I20260926 22:44:23.256178 2012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:44:23.257659 2012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:44:23 dist-test-slave-j8sq krb5kdc[1781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462663, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.252@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:44:23.264354 11695 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4c2jnj/build/release/bin/kudu as pid 1944
I20260926 22:44:23.264478 11695 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4c2jnj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462400949426-11695-0/minicluster-data/master-2/wal/instance
I20260926 22:44:23.264683 11695 external_mini_cluster.cc:1428] Setting key 1ab274f070d7b891595bff162d8fdc1a
I20260926 22:44:23.266968 1866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.258393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.11.107.254:35635)
0926 22:44:23.258461 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.258463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.258674 (+ 211us) server_negotiation.cc:407] Connection header received
0926 22:44:23.259097 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.259098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.259108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.259174 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.259768 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.260239 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.261157 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.261339 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.261399 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.263516 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.263517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.263517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.263524 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.265144 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.265729 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.265731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.265731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.265819 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.266198 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.266199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.266199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.266359 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.266443 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.266839 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.266901 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 22:44:23.267181 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.258366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.254:35635 (local address 127.0.0.1:41190)
0926 22:44:23.258513 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.258553 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:23.259003 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:23.259256 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:23.259259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:23.259494 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:23.259692 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.259696 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.260315 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.260315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.261079 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.261081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.261433 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.261434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.261538 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.262063 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:23.262071 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:23.263433 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:23.265249 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.265256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.265258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.265582 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.265950 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.265951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.265952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.266126 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.266476 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:23.266483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:23.266687 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 22:44:23.266816 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 22:44:23.266967 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260926 22:44:23.267796 2012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:44:23 dist-test-slave-j8sq krb5kdc[1781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.252@KRBTEST.COM
Sep 26 22:44:23 dist-test-slave-j8sq krb5kdc[1781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462663, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.252@KRBTEST.COM for kudu/127.11.107.253@KRBTEST.COM
I20260926 22:44:23.274487 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.265101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.11.107.252:44279)
0926 22:44:23.265258 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.265262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.265550 (+ 288us) server_negotiation.cc:407] Connection header received
0926 22:44:23.265574 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.265577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.265590 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.265689 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.266500 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.267518 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.268447 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.268802 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.268855 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.271483 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.271492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.271492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.271500 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.273175 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.273505 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.273508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.273508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.273567 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.273861 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.273862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.273862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.274046 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.274181 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.274286 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.274370 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":101,"threads_started":1}
I20260926 22:44:23.275033 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.268039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.253:33041 (local address 127.0.0.1:40554)
0926 22:44:23.268087 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.268103 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:23.268139 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:23.268266 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:23.268267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:23.268393 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:23.268466 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.268468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.269142 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.269143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.269748 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.269750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.270336 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.270337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.270407 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.270770 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:23.270776 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:23.272174 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:23.273971 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.273972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.273972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.274173 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.274379 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.274380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.274381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.274442 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.274746 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:23.274747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:23.274788 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 22:44:23.274850 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 22:44:23.274917 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":10}
I20260926 22:44:23.275074 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.268116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.11.107.253:33041)
0926 22:44:23.268167 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.268169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.268178 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:44:23.268202 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.268202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.268211 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.268256 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.268535 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.269068 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.269833 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.270243 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.270322 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.272274 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.272276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.272276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.272283 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.273892 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.274250 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.274252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.274252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.274310 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.274501 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.274502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.274502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.274655 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.274719 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.274889 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.274970 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:44:23.275477 2012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:44:23 dist-test-slave-j8sq krb5kdc[1781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462663, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.252@KRBTEST.COM for kudu/127.11.107.252@KRBTEST.COM
Sep 26 22:44:23 dist-test-slave-j8sq krb5kdc[1781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.254@KRBTEST.COM
I20260926 22:44:23.282377 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.275719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:44279 (local address 127.0.0.1:55760)
0926 22:44:23.275765 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.275774 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:23.275808 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:23.275935 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:23.275936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:23.276077 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:23.276111 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.276112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.276731 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.276732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.277199 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.277200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.277615 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.277616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.277667 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.278104 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:23.278108 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:23.279457 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:23.281276 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.281277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.281277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.281429 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.281676 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.281676 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.281676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.281724 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.282022 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:23.282022 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:23.282060 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 22:44:23.282150 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 22:44:23.282245 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 22:44:23.282385 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.275786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.11.107.252:44279)
0926 22:44:23.275819 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.275821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.275829 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:44:23.275851 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.275852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.275861 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.275930 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.276202 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.276640 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.277273 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.277552 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.277603 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.279542 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.279544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.279544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.279552 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.281193 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.281495 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.281497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.281497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.281554 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.281782 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.281783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.281783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.281951 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.282029 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.282171 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.282291 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":17}
I20260926 22:44:23.283560 1866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.275392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.11.107.254:35635)
0926 22:44:23.275433 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.275435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.275443 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:44:23.275466 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.275467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.275476 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.275533 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.276153 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.276628 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.277447 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.277670 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.277741 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.280289 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.280290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.280290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.280297 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.281739 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.282122 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.282123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.282124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.282188 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.282381 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.282382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.282383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.282531 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.283177 (+ 646us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.283364 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.283445 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:44:23.283764 2012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb: Bootstrap starting.
I20260926 22:44:23.284859 2012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:23.285171 2012 log.cc:824] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:23.286018 2012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb: No bootstrap required, opened a new log
I20260926 22:44:23.287672 2012 raft_consensus.cc:348] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } }
I20260926 22:44:23.287794 2012 raft_consensus.cc:729] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e81d70adea33483da821540ed3f93ebb, State: Initialized, Role: FOLLOWER
I20260926 22:44:23.287880 2012 consensus_queue.cc:260] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } }
I20260926 22:44:23.288187 2012 sys_catalog.cc:564] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:23.288182 2018 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } } }
I20260926 22:44:23.288282 2018 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 22:44:23 dist-test-slave-j8sq krb5kdc[1781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.107.253@KRBTEST.COM
W20260926 22:44:23.290125 2029 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:44:23.290177 2029 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:44:23.292332 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.284209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.11.107.253:33041)
0926 22:44:23.284258 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.284261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.284269 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:44:23.284292 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.284293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.284304 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.284390 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.284808 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.285519 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.286330 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.286620 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.286691 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.288990 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.288991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.288992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.288999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.290737 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.291104 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.291105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.291106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.291168 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.291384 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.291386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.291386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.291521 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.291706 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.292144 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.292223 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 22:44:23.290639 1944 master_runner.cc:186] Error getting master registration for 127.11.107.254:35635: OK, instance_id { permanent_uuid: "79701f22f64848a4ae0708896724d4be" instance_seqno: 1790462662962363 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 22:44:23.293089 1944 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 22:44:23.293905 11695 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:44:23.294806 1836 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "79701f22f64848a4ae0708896724d4be"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41206
I20260926 22:44:23.309115 1862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } attempt: 2
Sep 26 22:44:23 dist-test-slave-j8sq krb5kdc[1781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462662, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.254@KRBTEST.COM for kudu/127.11.107.252@KRBTEST.COM
I20260926 22:44:23.315836 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.309518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55772 (local address 127.11.107.252:44279)
0926 22:44:23.309586 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.309588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.309598 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:44:23.309650 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.309651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.309659 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.309781 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.310115 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.310578 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.311269 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.311455 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.311536 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.313301 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.313302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.313303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.313309 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.314709 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.315054 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.315055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.315055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.315108 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.315311 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.315312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.315312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.315449 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.315522 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.315678 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.315745 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 22:44:23.315852 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.309446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:44279 (local address 127.0.0.1:55772)
0926 22:44:23.309500 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.309512 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:23.309555 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:23.309807 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:23.309808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:23.309991 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:23.310032 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.310034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.310667 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.310668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.311169 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.311171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.311536 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.311537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.311595 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.312027 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:23.312032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:23.313206 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:23.314808 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.314809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.314810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.314982 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.315182 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.315183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.315183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.315240 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.315524 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:23.315525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:23.315570 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 22:44:23.315653 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 22:44:23.315766 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 22:44:23.317588 1862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be: Bootstrap starting.
I20260926 22:44:23.318347 1862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:23.318643 1862 log.cc:824] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:23.319371 1862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be: No bootstrap required, opened a new log
I20260926 22:44:23.320362 1862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } }
I20260926 22:44:23.320494 1862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79701f22f64848a4ae0708896724d4be, State: Initialized, Role: FOLLOWER
I20260926 22:44:23.320582 1862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } }
I20260926 22:44:23.320852 2032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } } }
I20260926 22:44:23.320914 1862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:23.320940 1836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 22:44:23.321046 1836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:23.321782 1836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } }
I20260926 22:44:23.322055 1836 leader_election.cc:288] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [CANDIDATE]: Term 1 election: Requested vote from peers 98ecfbbb31c649c7bb0e1bec7160cb57 (127.11.107.253:33041), e81d70adea33483da821540ed3f93ebb (127.11.107.252:44279)
I20260926 22:44:23.320968 2032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:44:23.323177 1986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79701f22f64848a4ae0708896724d4be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e81d70adea33483da821540ed3f93ebb"
I20260926 22:44:23.323314 1986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:23.324012 1986 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79701f22f64848a4ae0708896724d4be in term 1.
I20260926 22:44:23.324234 1811 leader_election.cc:302] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79701f22f64848a4ae0708896724d4be, e81d70adea33483da821540ed3f93ebb; no voters:
I20260926 22:44:23.324347 2032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:44:23.324468 2032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [term 1 LEADER]: Becoming Leader. State: Replica: 79701f22f64848a4ae0708896724d4be, State: Running, Role: LEADER
I20260926 22:44:23.324540 2032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } }
I20260926 22:44:23.324919 2032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79701f22f64848a4ae0708896724d4be. Latest consensus state: current_term: 1 leader_uuid: "79701f22f64848a4ae0708896724d4be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } } }
I20260926 22:44:23.324992 2032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [sys.catalog]: This master's current role is: LEADER
W20260926 22:44:23.325634 2043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:44:23.325712 2043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:44:23.325058 1794 master_runner.cc:186] Error getting master registration for 127.11.107.253:33041: OK, instance_id { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" instance_seqno: 1790462663106654 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 22:44:23.328004 1794 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 22:44:23.328707 2052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:44:23.329360 2052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:44:23.332670 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.324195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.11.107.253:33041)
0926 22:44:23.324352 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.324355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.324361 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:44:23.324380 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.324381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.324390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.324444 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.325935 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.326676 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.327735 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.328044 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.328085 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.329673 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.329675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.329675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.329682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.331373 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.331788 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.331789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.331789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.331843 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.332038 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.332039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.332039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.332194 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.332273 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.332502 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.332577 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260926 22:44:23.336604 1866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.325114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.11.107.254:35635)
0926 22:44:23.325147 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.325149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.325912 (+ 763us) server_negotiation.cc:407] Connection header received
0926 22:44:23.326004 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.326005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.326014 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.326041 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.328060 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.328765 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.331065 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.331364 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.331402 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.333831 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.333832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.333832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.333867 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.335441 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.335804 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.335805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.335805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.335854 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.336040 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.336040 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.336041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.336152 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.336238 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.336443 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.336515 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:44:23.338699 1986 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb [term 1 FOLLOWER]: Refusing update from remote peer 79701f22f64848a4ae0708896724d4be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:44:23.339033 2049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:44:23.341097 2018 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79701f22f64848a4ae0708896724d4be. Latest consensus state: current_term: 1 leader_uuid: "79701f22f64848a4ae0708896724d4be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } } }
I20260926 22:44:23.341197 2018 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:44:23.341691 1937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } attempt: 3
I20260926 22:44:23.343040 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.324902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.11.107.252:44279)
0926 22:44:23.324937 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.324939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.331022 (+ 6083us) server_negotiation.cc:407] Connection header received
0926 22:44:23.331290 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.331291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.331313 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.331342 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.332925 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.333686 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.335712 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.336019 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.336062 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.338776 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.338777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.338778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.338785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.340763 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.341799 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.341800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.341800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.341875 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.342338 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.342340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.342340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.342480 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.342573 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.342882 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.342956 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 26 22:44:23 dist-test-slave-j8sq krb5kdc[1781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462663, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.107.253@KRBTEST.COM for kudu/127.11.107.252@KRBTEST.COM
I20260926 22:44:23.351009 2049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79701f22f64848a4ae0708896724d4be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } } }
I20260926 22:44:23.351092 2049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:23.351260 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.341936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.107.252:44279 (local address 127.0.0.1:55786)
0926 22:44:23.341977 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 22:44:23.341987 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:44:23.342019 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:44:23.343293 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:44:23.343294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:44:23.343446 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:44:23.343485 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.343487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.344377 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.344378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.345160 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:44:23.345162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.345682 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.345683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:44:23.345760 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.346217 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:44:23.346235 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:44:23.347679 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:44:23.350023 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.350024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.350025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.350238 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.350508 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:44:23.350509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:44:23.350509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:44:23.350563 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:44:23.350974 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:44:23.350975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:44:23.351026 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 22:44:23.351082 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 22:44:23.351168 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 22:44:23.351775 2018 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79701f22f64848a4ae0708896724d4be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } } }
I20260926 22:44:23.351873 2018 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:44:23.351868 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:23.341961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.11.107.252:44279)
0926 22:44:23.343105 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.343106 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.343113 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:44:23.343131 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.343131 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.343140 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.343190 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.343579 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.344273 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.345268 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:23.345562 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:23.345622 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:23.347787 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:44:23.347789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:44:23.347789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:44:23.347796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:44:23.349891 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.350322 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.350323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.350324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.350399 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:44:23.350635 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:44:23.350636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:44:23.350636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:44:23.350852 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:44:23.351411 (+ 559us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:23.351478 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 22:44:23.351592 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1130,"thread_start_us":82,"threads_started":1}
I20260926 22:44:23.352372 2055 mvcc.cc:205] Tried to move back new op lower bound from 7333735069032923136 to 7333735068977905664. Current Snapshot: MvccSnapshot[applied={T|T < 7333735069032923136}]
I20260926 22:44:23.353354 1937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57: Bootstrap starting.
I20260926 22:44:23.354056 1937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57: Neither blocks nor log segments found. Creating new log.
I20260926 22:44:23.354341 1937 log.cc:824] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57: Log is configured to *not* fsync() on all Append() calls
I20260926 22:44:23.354660 2052 catalog_manager.cc:1347] Generated new cluster ID: 6efbc4231bca4d87b4659393450be9cb
I20260926 22:44:23.354763 2052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:44:23.355293 1937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57: No bootstrap required, opened a new log
I20260926 22:44:23.357242 1937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } }
I20260926 22:44:23.357362 1937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ecfbbb31c649c7bb0e1bec7160cb57, State: Initialized, Role: FOLLOWER
I20260926 22:44:23.357430 1937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } }
I20260926 22:44:23.357712 1937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:44:23.357774 1911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79701f22f64848a4ae0708896724d4be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57"
I20260926 22:44:23.357903 1910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:44:23.358693 1910 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57 [term 1 FOLLOWER]: Refusing update from remote peer 79701f22f64848a4ae0708896724d4be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:44:23.359083 2032 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [LEADER]: Connected to new peer: Peer: permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:44:23.361959 2062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79701f22f64848a4ae0708896724d4be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } } }
I20260926 22:44:23.362057 2063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79701f22f64848a4ae0708896724d4be. Latest consensus state: current_term: 1 leader_uuid: "79701f22f64848a4ae0708896724d4be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } } }
I20260926 22:44:23.362218 2063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:44:23.362321 2063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79701f22f64848a4ae0708896724d4be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } } }
I20260926 22:44:23.362413 2063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:44:23.362830 2064 mvcc.cc:205] Tried to move back new op lower bound from 7333735069032923136 to 7333735068977905664. Current Snapshot: MvccSnapshot[applied={T|T < 7333735069032923136}]
I20260926 22:44:23.363394 1911 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 79701f22f64848a4ae0708896724d4be for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260926 22:44:23.364507 2077 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:44:23.364612 2077 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:44:23.362077 2062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:44:23.371562 2032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79701f22f64848a4ae0708896724d4be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79701f22f64848a4ae0708896724d4be" member_type: VOTER last_known_addr { host: "127.11.107.254" port: 35635 } } peers { permanent_uuid: "98ecfbbb31c649c7bb0e1bec7160cb57" member_type: VOTER last_known_addr { host: "127.11.107.253" port: 33041 } } peers { permanent_uuid: "e81d70adea33483da821540ed3f93ebb" member_type: VOTER last_known_addr { host: "127.11.107.252" port: 44279 } } }
I20260926 22:44:23.371644 2032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be [sys.catalog]: This master's current role is: LEADER
I20260926 22:44:23.377939 2052 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:44:23.378644 2052 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:44:23.389909 2052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79701f22f64848a4ae0708896724d4be: Generated new TSK 0
I20260926 22:44:23.390096 2052 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 22:44:23.417168 1941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:23.415549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.11.107.253:33041)
0926 22:44:23.415594 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.415596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.416512 (+ 916us) server_negotiation.cc:407] Connection header received
0926 22:44:23.416618 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.416619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.416630 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.416661 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.417100 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38958: BlockingRecv error: recv got EOF from 127.0.0.1:38958 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 22:44:23.418100 1866 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:23.415933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.11.107.254:35635)
0926 22:44:23.415969 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.415971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.417588 (+ 1617us) server_negotiation.cc:407] Connection header received
0926 22:44:23.417681 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.417682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.417692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.417722 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.418038 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54148: BlockingRecv error: recv got EOF from 127.0.0.1:54148 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 22:44:23.418919 2016 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:23.416244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.11.107.252:44279)
0926 22:44:23.416282 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.416284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.418419 (+ 2135us) server_negotiation.cc:407] Connection header received
0926 22:44:23.418539 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.418540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.418550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.418579 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.418863 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54212: BlockingRecv error: recv got EOF from 127.0.0.1:54212 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 22:44:23.421281 1941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:23.420512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.11.107.253:33041)
0926 22:44:23.420544 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.420545 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.420768 (+ 223us) server_negotiation.cc:407] Connection header received
0926 22:44:23.420870 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.420871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.420888 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.420917 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.421217 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38968: BlockingRecv error: recv got EOF from 127.0.0.1:38968 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:44:23.424158 1941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:23.423426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.11.107.253:33041)
0926 22:44:23.423456 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.423457 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.423690 (+ 233us) server_negotiation.cc:407] Connection header received
0926 22:44:23.423781 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.423781 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.423791 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.423818 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.424113 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38978: BlockingRecv error: recv got EOF from 127.0.0.1:38978 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:44:23.427973 1941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:23.427251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.11.107.253:33041)
0926 22:44:23.427282 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.427284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.427516 (+ 232us) server_negotiation.cc:407] Connection header received
0926 22:44:23.427606 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.427607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.427616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.427642 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.427929 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38980: BlockingRecv error: recv got EOF from 127.0.0.1:38980 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:44:23.433810 1941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:23.433081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.11.107.253:33041)
0926 22:44:23.433113 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.433115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.433360 (+ 245us) server_negotiation.cc:407] Connection header received
0926 22:44:23.433448 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.433449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.433458 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.433484 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.433775 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38988: BlockingRecv error: recv got EOF from 127.0.0.1:38988 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:44:23.443861 1941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:23.443015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.11.107.253:33041)
0926 22:44:23.443051 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.443054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.443335 (+ 281us) server_negotiation.cc:407] Connection header received
0926 22:44:23.443436 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.443437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.443448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.443478 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.443814 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38990: BlockingRecv error: recv got EOF from 127.0.0.1:38990 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 22:44:23.462157 1941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:23.461210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.11.107.253:33041)
0926 22:44:23.461278 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.461280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.461557 (+ 277us) server_negotiation.cc:407] Connection header received
0926 22:44:23.461655 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.461656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.461667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.461697 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.462097 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39002: BlockingRecv error: recv got EOF from 127.0.0.1:39002 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 22:44:23.496285 1941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:23.495544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.11.107.253:33041)
0926 22:44:23.495592 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.495594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.495744 (+ 150us) server_negotiation.cc:407] Connection header received
0926 22:44:23.495822 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.495823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.495834 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.495907 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.496228 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39018: BlockingRecv error: recv got EOF from 127.0.0.1:39018 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 22:44:23.562315 1941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:23.561575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.11.107.253:33041)
0926 22:44:23.561627 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.561629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.561722 (+ 93us) server_negotiation.cc:407] Connection header received
0926 22:44:23.561767 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.561768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.561778 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.561897 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.562227 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39032: BlockingRecv error: recv got EOF from 127.0.0.1:39032 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 22:44:23.692278 1941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:23.691434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.11.107.253:33041)
0926 22:44:23.691503 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.691505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.691625 (+ 120us) server_negotiation.cc:407] Connection header received
0926 22:44:23.691731 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.691732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.691747 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.691819 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.692200 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39042: BlockingRecv error: recv got EOF from 127.0.0.1:39042 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 22:44:23.950344 1941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:44:23.949697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.11.107.253:33041)
0926 22:44:23.949748 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:23.949750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:23.949777 (+ 27us) server_negotiation.cc:407] Connection header received
0926 22:44:23.949807 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:23.949807 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:23.949825 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:23.949923 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:44:23.950298 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39044: BlockingRecv error: recv got EOF from 127.0.0.1:39044 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:44:24.290614 2029 catalog_manager.cc:1259] Loaded cluster ID: 6efbc4231bca4d87b4659393450be9cb
I20260926 22:44:24.290737 2029 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb: loading cluster ID for follower catalog manager: success
I20260926 22:44:24.291622 2029 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb: acquiring CA information for follower catalog manager: success
I20260926 22:44:24.291930 2029 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e81d70adea33483da821540ed3f93ebb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:44:24.365087 2077 catalog_manager.cc:1259] Loaded cluster ID: 6efbc4231bca4d87b4659393450be9cb
I20260926 22:44:24.365175 2077 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57: loading cluster ID for follower catalog manager: success
I20260926 22:44:24.365993 2077 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57: acquiring CA information for follower catalog manager: success
I20260926 22:44:24.366254 2077 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 98ecfbbb31c649c7bb0e1bec7160cb57: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:44:24.466295 2144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:24.463505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.11.107.253:33041)
0926 22:44:24.463632 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:24.463634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:24.463747 (+ 113us) server_negotiation.cc:407] Connection header received
0926 22:44:24.463842 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:24.463843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:24.463909 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:24.464005 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:44:24.464166 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:24.464673 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:24.465449 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:24.465657 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:24.465741 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:24.465937 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:44:24.466053 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:44:24.466153 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:24.466172 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 22:44:24.466193 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":53,"threads_started":1}
I20260926 22:44:24.470618 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:24.468204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.11.107.252:44279)
0926 22:44:24.468305 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:24.468307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:24.468320 (+ 13us) server_negotiation.cc:407] Connection header received
0926 22:44:24.468391 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:24.468392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:24.468425 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:24.468492 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:44:24.468673 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:24.469138 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:24.469895 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:24.470101 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:24.470163 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:24.470295 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:44:24.470384 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:44:24.470438 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:24.470490 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 22:44:24.470507 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
I20260926 22:44:24.475046 2144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:24.472231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.11.107.253:33041)
0926 22:44:24.472271 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:24.472273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:24.472379 (+ 106us) server_negotiation.cc:407] Connection header received
0926 22:44:24.472468 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:24.472469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:24.472481 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:24.472510 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:44:24.472684 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:24.473331 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:24.474208 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:24.474499 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:24.474568 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:24.474748 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:44:24.474788 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:44:24.474852 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:24.474929 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 22:44:24.474951 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:44:24.475514 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:24.472528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.11.107.252:44279)
0926 22:44:24.472567 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:24.472569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:24.472876 (+ 307us) server_negotiation.cc:407] Connection header received
0926 22:44:24.472951 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:24.472951 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:24.472961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:24.473008 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:44:24.473148 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:24.473804 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:24.474667 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:24.474944 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:24.475003 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:24.475194 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:44:24.475222 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:44:24.475265 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:24.475412 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 22:44:24.475434 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:44:24.475665 2156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:44:24.472232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.11.107.254:35635)
0926 22:44:24.472363 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 22:44:24.472365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:44:24.472537 (+ 172us) server_negotiation.cc:407] Connection header received
0926 22:44:24.472585 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:44:24.472586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:44:24.472660 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:44:24.472755 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:44:24.472853 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:24.473643 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:24.474672 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:44:24.474984 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:44:24.475022 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:44:24.475255 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:44:24.475351 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:44:24.475388 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 22:44:24.475512 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 22:44:24.475545 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260926 22:44:24.481557 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 1794
I20260926 22:44:24.487289 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 1869
I20260926 22:44:24.492345 11695 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4c2jnj/build/release/bin/kudu with pid 1944
2026-09-26T22:44:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2793 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5570 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (263548 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST